golang struct 或其他对象向 []byte 的序列化或反序列化

bytecodec 字节流编解码

这个库实现 struct 或其他对象向 []byte 的序列化或反序列化

可以帮助你在编写 tcp 服务,或者需要操作字节流时,简化数据的组包、解包

这个库的组织逻辑 copy 借鉴了标准库 encoding/json 🙏

安装

使用 go get 安装最新版本

go get -u github.com/lai323/bytecodec

然后在你的应用中导入

import "github.com/lai323/bytecodec"

使用

编码时 bytecodec 按照 struct 的字段顺序将字段一个个写入到 []byte 中;解码时根据字段类型读取对应长度的 byte 解析到字段中

嵌入字段和未导出字段会被忽略,也可以使用 bytecodec:"-" 标签,主动忽略一个字段

对于 int uint 被看作 64 位处理

对于空指针字段,编码时不会被忽略,会根据这个指针的类型创建一个空对象,写入到 []byte 中,所以当使用类似下面这种递归类型时,会返回错误,指示不支持这种类型

type s struct {
	Ptr1, Ptr2 *s
}

bytecodec 支持了可转为 byte 所有基础类型,结合下面的几个标签可以轻松的处理一般的字节数据的组包,解包

  • bytecodec:"length:5" 用于指定不能确定长度的类型的固定长度,对于 string 指的是字符串的字节长度,对于 slice 指的是元素个数,其他类型会忽略这个标签
  • bytecodec:"lengthref:FieldName" 用于控制不定长的数据,例如典型的,先从字节流中读取长度,在按这个长度读取后续数据
  • bytecodec:"gbk" bytecodec:"gbk18030" 用于为字符串类型指定编码格式
  • bytecodec:"bcd8421:5,true" 使用 BCD 压缩,第一个参数是压缩后 byte 长度,不足时在前面填充 0,第二个参数指示解码时,是否跳过首部的 0,这个标签应该使用在字符串类型的字段上,使用字符串表示数值,是为了处理较长的数字串

对于更加复杂的数据结构,你可以实现 bytecodec.ByteCoder 自定义编解码

type ByteCoder interface {
	MarshalBytes(*bytecodec.CodecState) error
	UnmarshalBytes(*bytecodec.CodecState) error
}

例子

", p.Header.SerialNo, p.Header.Time, p.Phone, p.MsgLength, p.Msg) } func marshal() { t := BCDTime(time.Date(2006, 01, 02, 15, 04, 05, 0, time.Local)) p := Packet{ Header: Header{ SerialNo: 1, Time: t, }, Phone: "18102169375", Msg: "你好", } b, err := bytecodec.Marshal(p) fmt.Println(fmt.Sprintf("%#v", b)) fmt.Println(err) } func unmarshal() { b := []byte{ 0x0, 0x1, 0x6, 0x1, 0x2, 0x15, 0x4, 0x5, 0x1, 0x81, 0x2, 0x16, 0x93, 0x75, 0x4, 0xc4, 0xe3, 0xba, 0xc3, } out := &Packet{} err := bytecodec.Unmarshal(b, out) fmt.Println(fmt.Sprintf("%v", out)) fmt.Println(err) } func main() { marshal() // []byte{ // 0x0, 0x1, // 0x6, 0x1, 0x2, 0x15, 0x4, 0x5, // 0x1, 0x81, 0x2, 0x16, 0x93, 0x75, // 0x4, // 0xc4, 0xe3, 0xba, 0xc3, // } // unmarshal() // // } ">
package main

import (
	"fmt"
	"time"

	"github.com/lai323/bcd8421"
	"github.com/lai323/bytecodec"
)

// 实现 bytecodec.ByteCoder 自定义时间字段的编解码
// 使用 BCD 压缩时间
var timeformat = "060102150405" // 2006-01-02 15:04:05
type BCDTime time.Time

func (bt BCDTime) MarshalBytes(cs *bytecodec.CodecState) error {
	tstr := bt.String()
	b, err := bcd8421.EncodeFromStr(tstr, 6)
	if err != nil {
		return err
	}
	cs.Write(b)
	return nil
}

func (bt *BCDTime) UnmarshalBytes(cs *bytecodec.CodecState) error {
	b := make([]byte, 6)
	cs.ReadFull(b)
	tstr, err := bcd8421.DecodeToStr(b, false)
	if err != nil {
		return err
	}

	t, err := time.ParseInLocation(timeformat, tstr, time.Local)
	if err != nil {
		return err
	}

	*bt = BCDTime(t)
	return nil
}

func (bt BCDTime) String() string {
	return time.Time(bt).Format(timeformat)
}

type Header struct {
	SerialNo uint16
	Time     BCDTime
}

type Packet struct {
	Header    Header
	Phone     string `bytecodec:"bcd8421:6,true"` // 使用长度为 6 的 BCD 8421 编码,解码时跳过数字前面的 0
	MsgLength uint8  `bytecodec:"lengthref:Msg"`  // 表示这个字段的值是 Msg 的字节长度
	Msg       string `bytecodec:"gbk"`            // 使用 GBK 编码
}

func (p Packet) String() string {
	return fmt.Sprintf("", p.Header.SerialNo, p.Header.Time, p.Phone, p.MsgLength, p.Msg)
}

func marshal() {
	t := BCDTime(time.Date(2006, 01, 02, 15, 04, 05, 0, time.Local))
	p := Packet{
		Header: Header{
			SerialNo: 1,
			Time:     t,
		},
		Phone: "18102169375",
		Msg:   "你好",
	}
	b, err := bytecodec.Marshal(p)
	fmt.Println(fmt.Sprintf("%#v", b))
	fmt.Println(err)
}

func unmarshal() {
	b := []byte{
		0x0, 0x1,
		0x6, 0x1, 0x2, 0x15, 0x4, 0x5,
		0x1, 0x81, 0x2, 0x16, 0x93, 0x75,
		0x4,
		0xc4, 0xe3, 0xba, 0xc3,
	}
	out := &Packet{}
	err := bytecodec.Unmarshal(b, out)
	fmt.Println(fmt.Sprintf("%v", out))
	fmt.Println(err)
}

func main() {
	marshal()
	// []byte{
	//     0x0, 0x1,
	//     0x6, 0x1, 0x2, 0x15, 0x4, 0x5,
	//     0x1, 0x81, 0x2, 0x16, 0x93, 0x75,
	//     0x4,
	//     0xc4, 0xe3, 0xba, 0xc3,
	// }
	// 

	unmarshal()
	// 
	// 
}
Similar Resources

A k-mer serialization package for Golang

A k-mer serialization package for Golang

.uniq v5 This package provides k-mer serialization methods for the package kmers, TaxIds of k-mers are optionally saved, while there's no frequency in

Aug 19, 2022

Some Golang types based on builtin. Implements interfaces Value / Scan and MarshalJSON / UnmarshalJSON for simple working with database NULL-values and Base64 encoding / decoding.

gotypes Some simple types based on builtin Golang types that implement interfaces for working with DB (Scan / Value) and JSON (Marshal / Unmarshal). N

Feb 12, 2022

gogoprotobuf is a fork of golang/protobuf with extra code generation features.

GoGo Protobuf looking for new ownership Protocol Buffers for Go with Gadgets gogoprotobuf is a fork of golang/protobuf with extra code generation feat

Nov 26, 2021

generic sort for slices in golang

slices generic sort for slices in golang basic API func BinarySearch[E constraints.Ordered](list []E, x E) int func IsSorted[E constraints.Ordered](li

Nov 3, 2022

Bitbank-trezor - Bitbank trezor with golang

Bitbank - Trezor (c) 2022 Bernd Fix [email protected] Y bitbank-trezor is fre

Jan 27, 2022

Kitex byte-dance internal Golang microservice RPC framework with high performance and strong scalability, customized extensions for byte internal.

Kitex byte-dance internal Golang microservice RPC framework with high performance and strong scalability, customized extensions for byte internal.

Kitex 字节跳动内部的 Golang 微服务 RPC 框架,具有高性能、强可扩展的特点,针对字节内部做了定制扩展。

Jan 9, 2023

Wrap byte read options with uniform interface for io.Reader and byte slice

nibbler Nibble chunks from Reader streams and slice in a common way Overview This is a golang module that provides an interface for treating a Reader

Dec 23, 2021

☄ The golang convenient converter supports Database to Struct, SQL to Struct, and JSON to Struct.

☄ The golang convenient converter supports Database to Struct, SQL to Struct, and JSON to Struct.

Gormat - Cross platform gopher tool The golang convenient converter supports Database to Struct, SQL to Struct, and JSON to Struct. 中文说明 Features Data

Dec 20, 2022

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

Pagser Pagser inspired by page parser。 Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and str

Dec 13, 2022

Copier for golang, copy value from struct to struct and more

Copier I am a copier, I copy everything from one to another Features Copy from field to field with same name Copy from method to field with same name

Jan 8, 2023

Shuffle bits in byte slice, written in golang

Shuffle 'em It is a library for bit shuffling, written in golang. The usecases a

Dec 27, 2021

goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configuration file.

goconfig goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configur

Dec 15, 2022

Match regex group into go struct using struct tags and automatic parsing

regroup Simple library to match regex expression named groups into go struct using struct tags and automatic parsing Installing go get github.com/oris

Nov 5, 2022

:100:Go Struct and Field validation, including Cross Field, Cross Struct, Map, Slice and Array diving

Package validator Package validator implements value validations for structs and individual fields based on tags. It has the following unique features

Jan 1, 2023

💯 Go Struct and Field validation, including Cross Field, Cross Struct, Map, Slice and Array diving

Package validator implements value validations for structs and individual fields based on tags.

Nov 9, 2022

Go generator to copy values from type to type and fields from struct to struct. Copier without reflection.

Copygen is a command-line code generator that generates type-to-type and field-to-field struct code without adding any reflection or dependenc

Dec 29, 2022

Membin is an in-memory database that can be stored on disk. Data model smiliar to key-value but values store as JSON byte array.

Membin Docs | Contributing | License What is Membin? The Membin database system is in-memory database smiliar to key-value databases, target to effici

Jun 3, 2021

ByNom is a Go package for parsing byte sequences, suitable for parsing text and binary data

ByNom is a Go package for parsing byte sequences. Its goal is to provide tools to build safe byte parsers without compromising the speed or memo

May 5, 2021

Bitwise AND on two byte-slices using SIMD instructions

This package provides a vectorised function which performs bitwise AND operation on all pairs of elements in two byte-slices. It detects CPU instruction set and chooses the available best one (AVX512, AVX2, SSE2).

Oct 17, 2022
Encode and decode Go (golang) struct types via protocol buffers.

protostructure protostructure is a Go library for encoding and decoding a struct type over the wire. This library is useful when you want to send arbi

Nov 15, 2022
Asn.1 BER and DER encoding library for golang.

WARNING This repo has been archived! NO further developement will be made in the foreseen future. asn1 -- import "github.com/PromonLogicalis/asn1" Pac

Nov 14, 2022
auto-generate capnproto schema from your golang source files. Depends on go-capnproto-1.0 at https://github.com/glycerine/go-capnproto

bambam: auto-generate capnproto schema from your golang source files. Adding capnproto serialization to an existing Go project used to mean writing a

Sep 27, 2022
Golang binary decoder for mapping data into the structure

binstruct Golang binary decoder to structure Install go get -u github.com/ghostiam/binstruct Examples ZIP decoder PNG decoder Use For struct From file

Dec 17, 2022
CBOR RFC 7049 (Go/Golang) - safe & fast with standard API + toarray & keyasint, CBOR tags, float64/32/16, fuzz tested.
CBOR RFC 7049 (Go/Golang) - safe & fast with standard API + toarray & keyasint, CBOR tags, float64/32/16, fuzz tested.

CBOR library in Go fxamacker/cbor is a CBOR encoder & decoder in Go. It has a standard API, CBOR tags, options for duplicate map keys, float64→32→16,

Jan 6, 2023
csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.
csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.

csvutil Package csvutil provides fast and idiomatic mapping between CSV and Go (golang) values. This package does not provide a CSV parser itself, it

Jan 6, 2023
Fixed width file parser (encoder/decoder) in GO (golang)

Fixed width file parser (encoder/decoder) for GO (golang) This library is using to parse fixed-width table data like: Name Address

Sep 27, 2022
Fast implementation of base58 encoding on golang.

Fast Implementation of Base58 encoding Fast implementation of base58 encoding in Go. Base algorithm is adapted from https://github.com/trezor/trezor-c

Dec 9, 2022
msgpack.org[Go] MessagePack encoding for Golang

MessagePack encoding for Golang ❤️ Uptrace.dev - All-in-one tool to optimize performance and monitor errors & logs Join Discord to ask questions. Docu

Dec 28, 2022
Dynamically Generates Ysoserial's Payload by Golang
Dynamically Generates Ysoserial's Payload by Golang

Gososerial 介绍 ysoserial是java反序列化安全方面著名的工具 无需java环境,无需下载ysoserial.jar文件 输入命令直接获得payload,方便编写安全工具 目前已支持CC1-CC7,K1-K4和CB1链 Introduce Ysoserial is a well-

Jul 10, 2022