A binary stream packer and unpacker

Last update: Dec 1, 2022

Comments: 7

binpacker

A binary packer and unpacker.

Install

go get github.com/zhuangsirui/binpacker

Examples

Packer

buffer := new(bytes.Buffer)
packer := binpacker.NewPacker(binary.BigEndian, buffer)
packer.PushByte(0x01)
packer.PushBytes([]byte{0x02, 0x03})
packer.PushUint16(math.MaxUint16)

// You can push data like this
buffer := new(bytes.Buffer)
packer := binpacker.NewPacker(binary.BigEndian, buffer)
packer.PushByte(0x01).PushBytes([]byte{0x02, 0x03}).PushUint16(math.MaxUint16)
packer.Error() // Make sure error is nil

Unpacker

Example data

buffer := new(bytes.Buffer)
packer := binpacker.NewPacker(binary.BigEndian, buffer)
unpacker := binpacker.NewUnpacker(binary.BigEndian, buffer)
packer.PushByte(0x01)
packer.PushUint16(math.MaxUint16)

var val1 byte
var val2 uint16
var err error
val1, err = unpacker.ShiftByte()
val2, err = unpacker.ShiftUint16()

var val1 byte
var val2 uint16
var err error
unpacker.FetchByte(&val1).FetchUint16(&val2)
unpacker.Error() // Make sure error is nil

Owner

Zhuang Sirui

Ruby, Rails, Golang

https://github.com/zhuangsirui/binpacker

Comments

Explicitly set byte order in new
I spent all day to find this problem :disappointed:. It was absolutely not obvious. https://github.com/zhuangsirui/binpacker/commit/f1b53c477754cc3d873952fd6bc1b759aa01a464#commitcomment-24315680

Byte order is only one important option when you need work with binary. There are sense to add it directly to signature.

I propose explicitly add byte order to new functions. It's go way. It should be obvious.

func NewPacker(writer io.Writer, byteOrder binary.ByteOrder) *Packer{} func NewUnpacker(reader io.Reader, byteOrder binary.ByteOrder) *Unpacker{}

I thought about NewPackerWithOrder but it's not good idea. binpacker single purpose package and it should be concise. SetByteOrder would be unused.

:warning:Comparability is broken. :warning: At least people will see immediately compilation error.
Typos in StringWithUint16Perfix, StringWithUint32Perfix etc.
Hello,

The following functions have typos in them. They should end in Prefix, not Perfix

StringWithUint16Perfix

StringWithUint32Perfix

StringWithUint64Perfix

BytesWithUint16Perfix

BytesWithUint32Perfix

BytesWithUint64Perfix
Semanting versioning

Hi! It was as 5 months when you proposed to use the semantic versioning. https://github.com/zhuangsirui/binpacker/issues/8#issuecomment-331052246

We Sturfeeinc use your binpacker in critical part of our system. I had big problems at previous time when I didn't use dependency management tools (#8). Now we have to be more responsible because we launch API soon.

Could you please to add version tags?

Recently I added versions tags to my library UTM.

Instruction

I propose 08a1b297435a414bec3ccf4215ff546dba418159 as v1.0.0

git checkout 08a1b297435a414bec3ccf4215ff546dba418159 .

Please change a message if you don't like it.

git tag -a v1.0.0 -m "Library was used in production for long time. We found one serious error. It considered stable now."

git push origin refs/tags/v1.0.0
Fix issue when io.Reader returns a partial read

First of all, thanks for great open source project. We use binpacker to unpack length prefixed data sent over network (using BytesWithUint32Prefix).

We found our data corrupted when we would be unpacking longer messages. It turned out to be an issue when io.Reader would return a partial read. The issue was caused here: buffer.Write(buf). If a read was smaller than allocated buf, data would be padded with a bunch of 0s at the end and all of it written to the final output buffer. The issue can be solved with a change to buffer.Write(buf[:rn]) but after I examined the method I thought it was better to leverage golang standard library instead. The bytes.Buffer usage was unnecessary since we know the final size in advance and we read it all anyway.

Let me know if you see any issues and I'll be happy to tweak this PR.
a typo in util.go

I believe there is a typo in util.go, I assume you've meant "prefixes" instead of "perfix"?

P.S. Thanks, this is a useful library, helps to avoid writing boilerplate.
support for int64

I see there is support for packing uint64. It will be really great if you could support int64.

May be there is a way to do it already and it is not documented? I see that you have worked on erlang :) How about implementing something like iolist_to_binary
gofmt
Just ran

gofmt -w .

on the project root. That's all.

https://blog.golang.org/go-fmt-your-code

I made this PR with a project going on over at https://github.com/rotblauer/gofmt-att, and it's definitely a work in progress. So if I got something wrong, or this is annoying at all, please file an issue over there and we'll sort it out.

Related tags

A Go implementation of a radix tree, that uses binary searches to speed up insert, retrieve and delete operations on dense trees

radixs A Go implementation of a radix tree, that uses binary searches to speed up insert, retrieve and delete operations on dense trees. This implemen

Feb 14, 2022

A Left-Leaning Red-Black (LLRB) implementation of balanced binary search trees for Google Go

GoLLRB GoLLRB is a Left-Leaning Red-Black (LLRB) implementation of 2-3 balanced binary search trees in Go Language. Overview As of this writing and to

Dec 23, 2022

Simple dense bitmap index in Go with binary operators

This package contains a bitmap index which is backed by uint64 slice, easily encodable to/from a []byte without copying memory around so it can be present in both disk and memory. As opposed to something as roaring bitmaps, this is a simple implementation designed to be used for small to medium dense collections.

Jan 3, 2023

Golang channel example with equivalent binary tree

golang_channel_example_with_equivalent_binary_tree Exercise: Equivalent Binary Trees There can be many different binary trees with the same sequence o

Oct 9, 2021

A project that deals with implementations of a binary tree

Binary Search Tree This is a project that deals with implementations of a binary tree and the following functions. Print Prints the entire tree. Argum

Nov 1, 2021

A slice backed binary heap with support for generic type parameters.

go-binaryheap A slice backed binary heap where the order can be customized by a comparison function. The main branch now requires go 1.18 because the

Jun 13, 2022

estruct traverses javascript projects and maps all the dependencies and relationships to a JSON. the output can be used to build network visualizations of the project and document the architecture.

EStruct traverses javascript projects and maps all the dependencies and relationships to a JSON. The output can be used to build network visualizations of the project and document the architecture.

Jan 27, 2022

Golang string comparison and edit distance algorithms library, featuring : Levenshtein, LCS, Hamming, Damerau levenshtein (OSA and Adjacent transpositions algorithms), Jaro-Winkler, Cosine, etc...

Go-edlib : Edit distance and string comparison library Golang string comparison and edit distance algorithms library featuring : Levenshtein, LCS, Ham

Dec 20, 2022

Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves.

Hilbert Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves. Documentation available here This is not an

Dec 23, 2022

Levenshtein distance and similarity metrics with customizable edit costs and Winkler-like bonus for common prefix.

A Go package for calculating the Levenshtein distance between two strings This package implements distance and similarity metrics for strings, based o

Dec 15, 2022

Decode / encode XML to/from map[string]interface{} (or JSON); extract values with dot-notation paths and wildcards. Replaces x2j and j2x packages.

mxj - to/from maps, XML and JSON Decode/encode XML to/from map[string]interface{} (or JSON) values, and extract/modify values from maps by key or key-

Dec 22, 2022

Dasel - Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool.

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

Jan 1, 2023

💯 Go Struct and Field validation, including Cross Field, Cross Struct, Map, Slice and Array diving

Package validator implements value validations for structs and individual fields based on tags.

Nov 9, 2022

Go translations of the algorithms and clients in the textbook Algorithms, 4th Edition by Robert Sedgewick and Kevin Wayne.

Overview Go translations of the Java source code for the algorithms and clients in the textbook Algorithms, 4th Edition by Robert Sedgewick and Kevin

Dec 13, 2022

Package iter provides generic, lazy iterators, functions for producing them from primitive types, as well as functions and methods for transforming and consuming them.

iter Package iter provides generic, lazy iterators, functions for producing them from primitive types, as well as functions and methods for transformi

Dec 16, 2022

A binary stream packer and unpacker

binpacker

Install

Examples

Packer

Unpacker

Owner

Zhuang Sirui

Comments

Explicitly set byte order in new

Typos in StringWithUint16Perfix, StringWithUint32Perfix etc.

Semanting versioning

Instruction

Fix issue when io.Reader returns a partial read

a typo in util.go

support for int64

gofmt