Encode and Decode Message Length Indicators for TCP/IP socket based protocols

SimpleMLI

A Message Length Indicator Encoder/Decoder

PkgGoDev Build Status Coverage Status Go Report Card

Message Length Indicators (MLI) are commonly used in communications over raw TCP/IP sockets. This method of denoting message length is especially popular with users of ISO 8583 messages, a common communication protocol for financial transactions.

This package provides an easy-to-use Encoder and Decoder for Message Length Indicators.

Usage:

import (
	"github.com/americanexpress/simplemli"
)

func main() {
	msg := []byte("This is a message")
	
	// Encoding Example
	mli, err := simplemli.Encode(simplemli.2I, len(msg))
	if err != nil {
		// Do something
	}
	
	// Append the MLI to the message
	msg := append(mli, msg)
	
	// Write to TCP Connection
	_, err = conn.Write(msg)
	
	
	// Reading MLI from TCP Connection
	b := make([]byte, simplemli.Size2I)
	_, err = conn.Read(&b) // only read the MLI from buffer
	if err != nil {
		// Do something
	}
	
	// Decoding Example
	length, err := simplemli.Decode(simplemli.2I, &b)
	if err != nil {
		// Do something
	}
	
	// Reading Message from TCP Connection
	msg := make([]byte, length)
	_, err = conn.Read(&msg)
}

Message Length Types

There are many common ways to encode message lengths and this library attempts to provide the most common MLI types.

Valid Options are listed in the table below.

Name Description
2I 2-byte network byte order with MLI included
2E 2-byte network byte order with MLI excluded
4I 4-byte network byte order with MLI included
4E 4-byte network byte order with MLI excluded
2EE 2-byte network byte order with MLI excluded, additional 2-byte header is included with message
2BCD2 2-byte Header with a 2-byte binary-coded decimal with MLI excluded
A4E 4-byte ASCII string with MLI excluded

Inclusive vs. Exclusive MLI

An inclusive MLI is an MLI type where the length of the Message Length Indicator itself is included in the MLI value.

For example, if a message is 1500 bytes and 2I encoded, the resulting MLI will have a value of 1502. The reverse is true for Exclusive MLI types, a 1500-byte message with a 2E encoded MLI will have a value of 1500.

When calling the Decoder, the MLI inclusive/exclusive nature is already taken care of. If you pass an MLI with a value of 1502 and decode it with 2I encoding. The resulting integer will be 1500.

Contributing

We welcome Your interest in the American Express Open Source Community on Github. Any Contributor to any Open Source Project managed by the American Express Open Source Community must accept and sign an Agreement indicating agreement to the terms below. Except for the rights granted in this Agreement to American Express and to recipients of software distributed by American Express, You reserve all right, title, and interest, if any, in and to Your Contributions. Please fill out the Agreement.

License

Any contributions made under this project will be governed by the Apache License 2.0.

Code of Conduct

This project adheres to the American Express Community Guidelines. By participating, you are expected to honor these guidelines.

Comments
  • Decode: use unsafe []byte->string conversion for discarded value

    Decode: use unsafe []byte->string conversion for discarded value

    For Decode's MLI4I, we invoke strconv.Atoi after string(*b) which incurs a []byte->string allocation. The converted []byte->string is discarded so anyways it is okay to use unsafe, as the string is never used again. You can see the results at https://dashboard.github.orijtech.com/benchmark/dbb0fca65f644d95bcf8d0c8caafc601

    Results:

    • time/op (ns/op) Encoding/Decoding_2I-8 7.5ns ± 1% 6.7ns ± 0% -10.36% (p=0.000 n=10+9) Encoding/Decoding_4E-8 8.8ns ± 0% 7.9ns ± 0% -10.35% (p=0.000 n=10+10) Encoding/Decoding_A4E-8 38ns ± 1% 17ns ± 0% -54.84% (p=0.000 n=10+10)

    • allocs/op (B/op) Encoding/Decoding_A4E-8 4.0B ± 0% 0 -100.00% (p=0.000 n=10+10)

    • allocs/op (N/op) Encoding/Decoding_A4E-8 1.0 ± 0% 0 -100.00% (p=0.000 n=10+10)

    Or visually Screen Shot 2021-09-25 at 12 58 38 AM

    Problem Statement

    What is the current behavior? Why and how does it need to change?

    string(*b) incurs a []byte->string allocation yet the value will be discarded.

    Description of Change

    Please include a summary of the change and, if applicable, tag related issues, add code snippets or logs.

    This change removes a []byte->string allocation

    Breaking Change

    Is this a breaking change?

    No.

    Caveats

    Please list any caveats or special considerations for this change.

    It uses unsafe, but the usage is isolated as we directly insert the converted string into strconv.Atoi, so this isn't a caveat really. /cc @kirbyquerby @cuonglm @madflojo

  • Improve performance by reducing conversions and allocation

    Improve performance by reducing conversions and allocation

    Avoid a hex.EncodeToString conversion just to compare against "00000000" and then later "0000", just by using bytes.Count(... '0')

    While here, also added (*testing.B).ReportAllocs to each benchmark to report allocations.

    Problem Statement

    What is the current behavior? Why and how does it need to change?

    Having examined this library's code and benchmarked it, we can reduce allocations by noticing that instead of invoking hex.EncodetoString "00000000" and then skip the if s == "0000" check and use string(*b) inside strconv.Atoi

    Description of Change

    Please include a summary of the change and, if applicable, tag related issues, add code snippets or logs.

    The benchmarking results https://dashboard.github.orijtech.com/benchmark/3319ffe57baa4553b37c704be8984aea show huge improvements in ~78% improvement in CPU time and also a huge reduction in allocations Screen Shot 2021-09-02 at 11 05 55 PM

    /cc @cuonglm @kirbyquerby

    Breaking Change

    Is this a breaking change?

    No

    Caveats

    Please list any caveats or special considerations for this change.

    None

  • Adding GitHub Actions for Lint and Tests

    Adding GitHub Actions for Lint and Tests

    Problem Statement

    What is the current behavior? Why and how does it need to change?

    Currently this repository is using Travis CI and due to some of the ways Travis CI has changed their support for Open Source projects it seems like it may be a challenge to use the Open Source tier in the future.

    Description of Change

    Please include a summary of the change and, if applicable, tag related issues, add code snippets or logs.

    This change adds two GitHub actions that will run on Pull Requests as well as the main/release branches. One will run make tests and the other will run the golangci-lint workflow.

    At this time, I'm not removing the Travis configuration, as these can run in conjunction.

    Breaking Change

    Is this a breaking change?

    No

    Caveats

    Please list any caveats or special considerations for this change.

  • Adding Badges for Travis, Coveralls, and Docs

    Adding Badges for Travis, Coveralls, and Docs

    Problem Statement

    What is the current behavior? Why and how does it need to change?

    Currently the Repository has no links to documentation, build status, coverage, or code quality.

    Description of Change

    Please include a summary of the change and, if applicable, tag related issues, add code snippets or logs.

    This change adds badges for all of the items above. It also has a gofmt change recommended by GoReportCard.

    Breaking Change

    Is this a breaking change?

    No

    Caveats

    Please list any caveats or special considerations for this change.

  • use constant name not value in documentation.

    use constant name not value in documentation.

    Problem Statement

    Documentation is out of sync with code. Uses constant value and not constant name.

    Description of Change

    Changes all instances in documentation from value to constant name.

    Breaking Change

    not

    Closes #5

Related tags
TCP output for beats to send events over TCP socket.

beats-tcp-output How To Use Clone this project to elastic/beats/libbeat/output/ Modify elastic/beats/libbeat/publisher/includes/includes.go : // add i

Aug 25, 2022
🚀Gev is a lightweight, fast non-blocking TCP network library based on Reactor mode. Support custom protocols to quickly and easily build high-performance servers.
🚀Gev is a lightweight, fast non-blocking TCP network library based on Reactor mode. Support custom protocols to quickly and easily build high-performance servers.

gev 中文 | English gev is a lightweight, fast non-blocking TCP network library based on Reactor mode. Support custom protocols to quickly and easily bui

Jan 6, 2023
Glue - Robust Go and Javascript Socket Library (Alternative to Socket.io)

Glue - Robust Go and Javascript Socket Library Glue is a real-time bidirectional socket library. It is a clean, robust and efficient alternative to so

Nov 25, 2022
GOWS is GoLang web-socket module Provides you with ease of handling web socket connections with a few lines

GOWS GOWS is GoLang web-socket module Provides you with ease of handling web socket connections with a few lines, it supports multi-connection on one

Apr 4, 2022
A TCP socket based chat server implemented using Go

Go Chat Server A better TCP socket chat server implemented using Go Connecting nc localhost 5000 Docker Build the container image docker build -t grub

Oct 16, 2021
Bee is a tool to scan ports by TCP and UDP protocols

Bee - Port scan tool ?? Bee is a tool to scan ports by TCP and UDP protocols Building from Source Code First, we compile the source code with the ligh

Oct 10, 2021
Golang pow implementation client <-> server over UDP and TCP protocols
Golang pow implementation client <-> server over UDP and TCP protocols

Client <-> server over UDP and TCP pow protocol Denial-of-Service-attacks are a typical situation when providing services over a network. A method for

Jan 13, 2022
Run commands on remote hosts, inspecting key indicators to manage infrastructure

inspector This is a very basic ssh helper tool to manage a smaller (few 100s up to a few 1000s) fleet of servers. The main point of inspector is to pr

Mar 3, 2022
Provide cloud-edge message synergy solutions for companies and individuals.the cloud-edge message system based on NATS.

Swarm This project is a cloud-edge synergy solution based on NATS. quikly deploy cloud deploy on k8s #pull the project. git clone https://github.com/g

Jan 11, 2022
TcpRoute , TCP 层的路由器。对于 TCP 连接自动从多个线路(电信、联通、移动)、多个域名解析结果中选择最优线路。

TcpRoute2 TcpRoute , TCP 层的路由器。对于 TCP 连接自动从多个线路(允许任意嵌套)、多个域名解析结果中选择最优线路。 TcpRoute 使用激进的选路策略,对 DNS 解析获得的多个IP同时尝试连接,同时使用多个线路进行连接,最终使用最快建立的连接。支持 TcpRoute

Dec 27, 2022
Multiplexer over TCP. Useful if target server only allows you to create limited tcp connections concurrently.

tcp-multiplexer Use it in front of target server and let your client programs connect it, if target server only allows you to create limited tcp conne

May 27, 2021
Tcp chat go - Create tcp chat in golang

TCP chat in GO libs Go net package and goroutines and channels tcp tcp or transm

Feb 5, 2022
golibwireshark - Package use libwireshark library to decode pcap file and analyse dissection data.

golibwireshark Package golibwireshark use libwireshark library to decode pcap file and analyse dissection data. This package can only be used in OS li

Nov 26, 2022
Implementation of Minecraft protocols : ping, query and icon.
Implementation of Minecraft protocols : ping, query and icon.

mcutils - Implementation of Minecraft protocols in Go Informations General All protocols are implemented in Go, without any external dependency. All p

Dec 19, 2022
Package for downloading things from a string URL using a variety of protocols.

go-getter is a library for Go (golang) for downloading files or directories from various sources using a URL as the primary form of input.

Jan 6, 2023
Use pingser to create client and server based on ICMP Protocol to send and receive custom message content.
Use pingser to create client and server based on ICMP Protocol to send and receive custom message content.

pingser Use pingser to create client and server based on ICMP Protocol to send and receive custom message content. examples source code: ./examples Us

Nov 9, 2022
Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and deadline support. MIT Licensed.

socket Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and d

Dec 14, 2022
Native macOS networking for QEMU using vmnet.framework and socket networking.

qemu-vmnet Native macOS networking for QEMU using vmnet.framework and socket networking. Getting started TODO -netdev socket,id=net0,udp=:1234,localad

Jan 5, 2023