A blazingly fast JSON serializing & deserializing library

Sonic

A blazingly fast JSON serializing & deserializing library, accelerated by JIT(just-in-time compiling) and SIMD(single-instruction-multi-data).

Benchmarks

For all sizes of json and all scenes of usage, Sonic performs almost best.

  • Small (400B, 11 keys, 3 levels) small benchmarks
  • Medium (110KB, 300+ keys, 3 levels, with many quoted-json values) medium benchmarks
  • Large (550KB, 10000+ key, 6 levels) large benchmarks

For a 13KB TwitterJson(cpu i9-9880H, goarch amd64), Sonic is 1.5x faster than json-iterator in decoding, 2.5x faster in encoding.

BenchmarkDecoder_Generic_Sonic-16                          10000             54309 ns/op         240.01 MB/s       46149 B/op        303 allocs/op
BenchmarkDecoder_Generic_StdLib-16                         10000            135268 ns/op          96.36 MB/s       50899 B/op        772 allocs/op
BenchmarkDecoder_Generic_JsonIter-16                       10000             96701 ns/op         134.80 MB/s       55791 B/op       1068 allocs/op
BenchmarkDecoder_Binding_Sonic-16                          10000             29478 ns/op         442.20 MB/s       26062 B/op         34 allocs/op
BenchmarkDecoder_Binding_StdLib-16                         10000            119348 ns/op         109.22 MB/s       10560 B/op        207 allocs/op
BenchmarkDecoder_Binding_JsonIter-16                       10000             37646 ns/op         346.25 MB/s       14673 B/op        385 allocs/op
BenchmarkEncoder_Generic_Sonic-16                          10000             25894 ns/op         503.39 MB/s       19096 B/op         42 allocs/op
BenchmarkEncoder_Generic_JsonIter-16                       10000             50275 ns/op         259.27 MB/s       13432 B/op         77 allocs/op
BenchmarkEncoder_Generic_StdLib-16                         10000            154901 ns/op          84.15 MB/s       48173 B/op        827 allocs/op
BenchmarkEncoder_Binding_Sonic-16                          10000              7373 ns/op        1768.04 MB/s       13861 B/op          4 allocs/op
BenchmarkEncoder_Binding_JsonIter-16                       10000             23223 ns/op         561.31 MB/s        9489 B/op          2 allocs/op
BenchmarkEncoder_Binding_StdLib-16                         10000             19512 ns/op         668.07 MB/s        9477 B/op          1 allocs/op

More detail see ast/search_test.go, decoder/decoder_test.go, encoder/encoder_test.go,

Usage

Marshal/Unmarshal

The behaviors are mostly consistent with encoding/json, except some uncommon escaping and key sorting (see issue4)

import "github.com/bytedance/sonic"

// Marshal
output, err := sonic.Marshal(&data) 
// Unmarshal
err := sonic.Unmarshal(input, &data) 

Get

Search partial json by given pathes, which must be non-negative integer or string or nil

import "github.com/bytedance/sonic"

input := []byte(`{"key1":[{},{"key2":{"key3":[1,2,3]}}]}`)

// no path, returns entire json
root, err := sonic.Get(input)
raw := root.Raw() // == string(input)

// multiple pathes
root, err := sonic.Get(input, "key1", 1, "key2")
sub := root.Get("key3").Index(2).Int64() // == 3

Returned ast.Node supports:

  • secondary search: Get(), Index(), GetByPath()
  • type assignment: Int64(), Float64(), String(), Number(), Bool(), Map(), Array()
  • children traversal: Values(), Properties()
  • supplement: Set(), SetByIndex(), Add(), Cap(), Len()

Use Number/Use Int64

import "github.com/bytedance/sonic/decoder"

input := `1`
var data interface{}

// default float64
dc := decoder.NewDecoder(input) 
dc.Decode(&data) // data == float64(1)
// use json.Number
dc = decoder.NewDecoder(input)
dc.UseNumber()
dc.Decode(&data) // data == json.Number("1")
// use int64
dc = decoder.NewDecoder(input)
dc.UseInt64()
dc.Decode(&data) // data == int64(1)

root, err := sonic.GetFromString(input)
// Get json.Number
jn := root.Number()
jm := root.InterfaceUseNumber().(json.Number) // jn == jm
// Get float64
fn := root.Float64()
fm := root.Interface().(float64) // jn == jm

Tips

Pretouch

Since Sonic uses JIT(just-in-time) compiling for decoder/encoder, huge schema may cause request-timeout. For better stability, we suggest to use Pretouch() for more-than-10000-field schema(struct) before Marshal()/Unmarshal().

import (
    "reflect"
    "github.com/bytedance/sonic"
)

func init() {
    var v HugeStruct
    err := sonic.Pretouch(reflect.TypeOf(v))
}

Pass string or []byte?

For alignment to encoding/json, we provide API to pass []byte as arguement, but the string-to-bytes copy is conducted at the same time considering safety, which may lose performance when origin json is huge. Therefore, you can use UnmarshalString, GetFromString to pass string, as long as your origin data is string or nocopy-cast is safe for your []byte.

Avoid repeating work

Get() overlapping pathes from the same root may cause repeating parsing. Instead of using Get() several times, you can use parser and searcher together like this:

import "github.com/bytedance/sonic"

root, err := sonic.GetByString(_TwitterJson, "statuses", 3, "user")
a = root.GetByPath( "entities","description")
b = root.GetByPath( "entities","url")
c = root.GetByPath( "created_at")

No need to worry about the overlaping or overparsing of a, b and c, because the inner parser of their root is lazy-loaded.

Better performance for generic deserializing

In most cases of fully-load generic json, Unmarshal() performs better than ast.Loads(). But if you only want to search a partial json and convert it into interface{} (or map[string]interface{}, []interface{}), we advise you to combine Get() and Unmarshal():

import "github.com/bytedance/sonic"

node, err := sonic.GetByString(_TwitterJson, "statuses", 3, "user")
var user interface{}
err = sonic.UnmarshalString(node.Raw(), &user)
Owner
Comments
  • Lefting unnecessary `json.Unmarshaler`/`json.Marshaler` implementation on main struct may introduce significant performance decline (ex: using easyjson code-gen)

    Lefting unnecessary `json.Unmarshaler`/`json.Marshaler` implementation on main struct may introduce significant performance decline (ex: using easyjson code-gen)

    `goos: darwin
    goarch: amd64
    pkg: gateway/test
    cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
    BenchmarkTestUnJsoniter-4         100000              2211 ns/op            4672 B/op         10 allocs/op
    BenchmarkTestUnJsoniter-4         100000              2106 ns/op            4672 B/op         10 allocs/op
    BenchmarkTestUnJsoniter-4         100000              1514 ns/op            4672 B/op         10 allocs/op
    BenchmarkTestUnJsoniter-4         100000              1447 ns/op            4672 B/op         10 allocs/op
    BenchmarkTestUnSonicJson-4        100000             31926 ns/op           14922 B/op        238 allocs/op
    BenchmarkTestUnSonicJson-4        100000             30677 ns/op           14907 B/op        238 allocs/op
    BenchmarkTestUnSonicJson-4        100000             30704 ns/op           14922 B/op        238 allocs/op
    BenchmarkTestUnSonicJson-4        100000             31479 ns/op           14905 B/op        238 allocs/op
    BenchmarkTestJsoniter-4           100000             20541 ns/op            7176 B/op         38 allocs/op
    BenchmarkTestJsoniter-4           100000             21243 ns/op            7176 B/op         38 allocs/op
    BenchmarkTestJsoniter-4           100000             20656 ns/op            7176 B/op         38 allocs/op
    BenchmarkTestJsoniter-4           100000             21710 ns/op            7176 B/op         38 allocs/op
    BenchmarkTestSonic-4              100000             26187 ns/op            7314 B/op         41 allocs/op
    BenchmarkTestSonic-4              100000             26092 ns/op            7314 B/op         41 allocs/op
    BenchmarkTestSonic-4              100000             25784 ns/op            7298 B/op         41 allocs/op
    BenchmarkTestSonic-4              100000             26414 ns/op            7377 B/op         41 allocs/op
    
  • Would you please add goccy/go-json as a benchmark target ?

    Would you please add goccy/go-json as a benchmark target ?

    I develop goccy/go-json. Since this library has same concept of yours about compatibility with encoding/json and high performance, I would be grateful if you could add it to the benchmark target.

  • sonic/decoder.StreamDecoder return unexpected SyntaxError

    sonic/decoder.StreamDecoder return unexpected SyntaxError

    Issue

    When using a StreamDecoder, I run into unexpected syntax error that neither go-json nor encoding/json seems to detect. I try to Pretouch the struct (you never know) but didn't change a thing. It seems like no option could resolve this.

    Not that great solution

    I manage to make it work by using a sonic.Decoder and a scanner like so

    var (
        dec sonic.Decoder
        input string
        record MyStruct
    )
    scanner := bufio.NewScanner(os.Stdin)
    
    for scanner.Scan() {
        input = scanner.Text()
        dec.Reset(input)
        dec.Decode(&record)
        // ....
    }
    

    But it makes the package way slower than go-json where it should be alot faster.

    Solutions ?

    Is there a way to enforce my StreamDecoder to skip this value and goto next one ?

  • 反序列化结构中的字符串是不安全的

    反序列化结构中的字符串是不安全的

    import (
    	"bytes"
    	"encoding/json"
    	"testing"
    
    	"github.com/bytedance/sonic"
    	jsoniter "github.com/json-iterator/go"
    	"github.com/stretchr/testify/require"
    )
    
    func TestSonicUnmarshal(t *testing.T) {
    	type test struct {
    		Str1 string
    		Str2 string
    	}
    	buf := bytes.NewBuffer(make([]byte, 0, 4096))
    	data, _ := sonic.Marshal(test{"1234567", "7654321"})
    	buf.Write(data)
    	var s test
    	sonic.Unmarshal(buf.Bytes(), &s)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    	buf.Reset()
    	data, _ = sonic.Marshal(test{"7654321", "1234567"})
    	buf.Write(data)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    }
    
    func TestStdJsonUnmarshal(t *testing.T) {
    	type test struct {
    		Str1 string
    		Str2 string
    	}
    	buf := bytes.NewBuffer(make([]byte, 0, 4096))
    	data, _ := json.Marshal(test{"1234567", "7654321"})
    	buf.Write(data)
    	var s test
    	json.Unmarshal(buf.Bytes(), &s)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    	buf.Reset()
    	data, _ = json.Marshal(test{"7654321", "1234567"})
    	buf.Write(data)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    }
    
    func TestJsoniterUnmarshal(t *testing.T) {
    	type test struct {
    		Str1 string
    		Str2 string
    	}
    	json := jsoniter.ConfigFastest
    	buf := bytes.NewBuffer(make([]byte, 0, 4096))
    	data, _ := json.Marshal(test{"1234567", "7654321"})
    	buf.Write(data)
    	var s test
    	json.Unmarshal(buf.Bytes(), &s)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    	buf.Reset()
    	data, _ = json.Marshal(test{"7654321", "1234567"})
    	buf.Write(data)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    }
    

    如果把 sonic 换成 json 或者 jsoniter,都能测试通过,而 sonic 则会报错,因为它反序列化结果中的 string 字段会直接引用输入参数中的值,如果在反序列化之后,输入参数被修改,则会导致反序列化结果错乱。

  • Not 100% compatibility with encoding/json

    Not 100% compatibility with encoding/json

    func TestCompatibility(t *testing.T) {
    	inputs := []interface{}{`"<&>"`, `"\"<&>\""`, "\b", float64(-0), float32(-0), map[string]int{"3": 3, "2": 2, "1": 1}}
    	for _, input := range inputs {
    		t.Run(fmt.Sprintf("case %v", input), func(t *testing.T) {
    			buf1, err1 := json.Marshal(input)
    			buf2, err2 := Marshal(input)
    			require.Nil(t, err1)
    			require.Nil(t, err2)
    			require.Equal(t, string(buf1), string(buf2))
    		})
    	}
    }
    
    --- FAIL: TestCompatibility (0.00s)
        --- FAIL: TestCompatibility/case_"<&>" (0.00s)
            fuzz_test.go:112: 
                    Error Trace:    fuzz_test.go:112
                    Error:          Not equal: 
                                    expected: "\"\\\"\\u003c\\u0026\\u003e\\\"\""
                                    actual  : "\"\\\"<&>\\\"\""
                                
                                    Diff:
                                    --- Expected
                                    +++ Actual
                                    @@ -1 +1 @@
                                    -"\"\u003c\u0026\u003e\""
                                    +"\"<&>\""
                    Test:           TestCompatibility/case_"<&>"
        --- FAIL: TestCompatibility/case_"\"<&>\"" (0.00s)
            fuzz_test.go:112: 
                    Error Trace:    fuzz_test.go:112
                    Error:          Not equal: 
                                    expected: "\"\\\"\\\\\\\"\\u003c\\u0026\\u003e\\\\\\\"\\\"\""
                                    actual  : "\"\\\"\\\\\\\"<&>\\\\\\\"\\\"\""
                                
                                    Diff:
                                    --- Expected
                                    +++ Actual
                                    @@ -1 +1 @@
                                    -"\"\\\"\u003c\u0026\u003e\\\"\""
                                    +"\"\\\"<&>\\\"\""
                    Test:           TestCompatibility/case_"\"<&>\""
        --- FAIL: TestCompatibility/case_\b (0.00s)
            fuzz_test.go:112: 
                    Error Trace:    fuzz_test.go:112
                    Error:          Not equal: 
                                    expected: "\"\\u0008\""
                                    actual  : "\"\\b\""
                                
                                    Diff:
                                    --- Expected
                                    +++ Actual
                                    @@ -1 +1 @@
                                    -"\u0008"
                                    +"\b"
                    Test:           TestCompatibility/case_\b
        --- FAIL: TestCompatibility/case_map[1:1_2:2_3:3] (0.00s)
            fuzz_test.go:112: 
                    Error Trace:    fuzz_test.go:112
                    Error:          Not equal: 
                                    expected: "{\"1\":1,\"2\":2,\"3\":3}"
                                    actual  : "{\"2\":2,\"1\":1,\"3\":3}"
                                
                                    Diff:
                                    --- Expected
                                    +++ Actual
                                    @@ -1 +1 @@
                                    -{"1":1,"2":2,"3":3}
                                    +{"2":2,"1":1,"3":3}
                    Test:           TestCompatibility/case_map[1:1_2:2_3:3]
    FAIL
    exit status 1
    FAIL    github.com/bytedance/sonic      0.144s
    
  • bug: go test panic in /encoder and /decoder

    bug: go test panic in /encoder and /decoder

    overview

    The unit test panicked in /encoder and /decoder, saying :

    # github.com/goccy/go-json/internal/encoder/vm 
    fatal error: runtime: out of memory
    

    go version

    go version go1.16 linux/amd64

    go env

    GO111MODULE=""
    GOARCH="amd64"
    GOBIN="/home/liwm29/gopath/bin"
    GOCACHE="/home/liwm29/.cache/go-build"
    GOENV="/home/liwm29/.config/go/env"
    GOEXE=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOINSECURE=""
    GOMODCACHE="/home/liwm29/gopath/pkg/mod"
    GONOPROXY=""
    GONOSUMDB=""
    GOOS="linux"
    GOPATH="/home/liwm29/gopath"
    GOPRIVATE=""
    GOPROXY="https://goproxy.cn,direct"
    GOROOT="/usr/local/go"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
    GOVCS=""
    GOVERSION="go1.16"
    GCCGO="gccgo"
    AR="ar"
    CC="gcc"
    CXX="g++"
    CGO_ENABLED="1"
    GOMOD="/home/liwm29/sonic_raw/sonic/go.mod"
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1917513849=/tmp/go-build -gno-record-gcc-switches"
    

    what happened

    [liwm29@VM-12-11-centos sonic]$ go test ./...
    ok      github.com/bytedance/sonic      4.103s
    ok      github.com/bytedance/sonic/ast  0.006s
    # github.com/goccy/go-json/internal/encoder/vm
    fatal error: runtime: out of memory
    
    runtime stack:
    runtime.throw(0xcac67f, 0x16)
            /usr/local/go/src/runtime/panic.go:1117 +0x72
    runtime.sysMap(0xc030000000, 0xc000000, 0x11e7b90)
            /usr/local/go/src/runtime/mem_linux.go:169 +0xc6
    runtime.(*mheap).sysAlloc(0x11c8bc0, 0xa000000, 0x42bcf7, 0x11c8bc8)
            /usr/local/go/src/runtime/malloc.go:729 +0x1e5
    runtime.(*mheap).grow(0x11c8bc0, 0x4ef7, 0x0)
            /usr/local/go/src/runtime/mheap.go:1346 +0x85
    runtime.(*mheap).allocSpan(0x11c8bc0, 0x4ef7, 0x440100, 0x7f1fa3d21688)
            /usr/local/go/src/runtime/mheap.go:1173 +0x609
    runtime.(*mheap).alloc.func1()
            /usr/local/go/src/runtime/mheap.go:910 +0x59
    runtime.systemstack(0x46d334)
            /usr/local/go/src/runtime/asm_amd64.s:379 +0x66
    runtime.mstart()
            /usr/local/go/src/runtime/proc.go:1246
    
    goroutine 1 [running]:
    runtime.systemstack_switch()
            /usr/local/go/src/runtime/asm_amd64.s:339 fp=0xc00286cac0 sp=0xc00286cab8 pc=0x46d460
    runtime.(*mheap).alloc(0x11c8bc0, 0x4ef7, 0xc002860001, 0xc00458c000)
            /usr/local/go/src/runtime/mheap.go:904 +0x85 fp=0xc00286cb10 sp=0xc00286cac0 pc=0x4279a5
    runtime.(*mcache).allocLarge(0x7f1fcac23108, 0x9dee000, 0xc002be0100, 0x7f1fa3d21688)
            /usr/local/go/src/runtime/mcache.go:224 +0x97 fp=0xc00286cb68 sp=0xc00286cb10 pc=0x418017
    runtime.mallocgc(0x9dee000, 0x0, 0x0, 0xc002be2000)
            /usr/local/go/src/runtime/malloc.go:1078 +0x925 fp=0xc00286cbf0 sp=0xc00286cb68 pc=0x40de65
    runtime.growslice(0xc686a0, 0xc025b74000, 0x7e5800, 0x7e5800, 0x7e5801, 0x1122970, 0x1, 0xc0)
            /usr/local/go/src/runtime/slice.go:224 +0x154 fp=0xc00286cc58 sp=0xc00286cbf0 pc=0x44fcd4
    cmd/compile/internal/ssa.(*debugState).appendLiveSlot(...)
            /usr/local/go/src/cmd/compile/internal/ssa/debug.go:290
    cmd/compile/internal/ssa.(*debugState).liveness(0xc001ccd660, 0xc0014b33b0, 0xc0032bb030, 0xc0047a58d8)
            /usr/local/go/src/cmd/compile/internal/ssa/debug.go:512 +0x7e5 fp=0xc00286ce58 sp=0xc00286cc58 pc=0x6705a5
    cmd/compile/internal/ssa.BuildFuncDebug(0xc0000fa000, 0xc0018c51e0, 0xc0041be300, 0xcc81f8, 0x6a9)
            /usr/local/go/src/cmd/compile/internal/ssa/debug.go:436 +0xa65 fp=0xc00286d0d0 sp=0xc00286ce58 pc=0x66f345
    cmd/compile/internal/gc.genssa(0xc0018c51e0, 0xc004c62000)
            /usr/local/go/src/cmd/compile/internal/gc/ssa.go:6527 +0x28ac fp=0xc00286d5b0 sp=0xc00286d0d0 pc=0xb437ec
    cmd/compile/internal/gc.compileSSA(0xc0000e5e40, 0x0)
            /usr/local/go/src/cmd/compile/internal/gc/pgen.go:329 +0x3a5 fp=0xc00286d690 sp=0xc00286d5b0 pc=0xad7685
    cmd/compile/internal/gc.compile(0xc0000e5e40)
            /usr/local/go/src/cmd/compile/internal/gc/pgen.go:277 +0x39e fp=0xc00286d708 sp=0xc00286d690 pc=0xad711e
    cmd/compile/internal/gc.funccompile(0xc0000e5e40)
            /usr/local/go/src/cmd/compile/internal/gc/pgen.go:220 +0xc5 fp=0xc00286d760 sp=0xc00286d708 pc=0xad6c65
    cmd/compile/internal/gc.Main(0xcc7d20)
            /usr/local/go/src/cmd/compile/internal/gc/main.go:762 +0x3525 fp=0xc00286df10 sp=0xc00286d760 pc=0xaabbe5
    main.main()
            /usr/local/go/src/cmd/compile/main.go:52 +0xb1 fp=0xc00286df88 sp=0xc00286df10 pc=0xbf90d1
    runtime.main()
            /usr/local/go/src/runtime/proc.go:225 +0x256 fp=0xc00286dfe0 sp=0xc00286df88 pc=0x439d76
    runtime.goexit()
            /usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00286dfe8 sp=0xc00286dfe0 pc=0x46f161
    FAIL    github.com/bytedance/sonic/decoder [build failed]
    FAIL    github.com/bytedance/sonic/encoder [build failed]
    ok      github.com/bytedance/sonic/internal/caching     0.002s
    ?       github.com/bytedance/sonic/internal/cpu [no test files]
    ok      github.com/bytedance/sonic/internal/jit 0.004s
    ok      github.com/bytedance/sonic/internal/loader      0.003s
    ?       github.com/bytedance/sonic/internal/native      [no test files]
    ok      github.com/bytedance/sonic/internal/native/avx  0.006s
    ok      github.com/bytedance/sonic/internal/native/avx2 0.006s
    ?       github.com/bytedance/sonic/internal/native/types        [no test files]
    ok      github.com/bytedance/sonic/internal/resolver    0.002s
    ?       github.com/bytedance/sonic/internal/rt  [no test files]
    ?       github.com/bytedance/sonic/unquote      [no test files]
    FAIL
    

    other

    i have changed nothing, except cloning and cd /sonic and run go test ./...

  • sonic.Unmarshal decode json string failed when field's type is

    sonic.Unmarshal decode json string failed when field's type is "stringint"

    test code below:

    type P struct {
    	Cip        string      `json:"cip"`
    	Oip        string      `json:"oip"`
    	Oid        string      `json:"oid"`
    	Aid        json.Number `json:"aid"`
    	Pcid       string      `json:"pcid"`
            Num        int `json:"num"`
    }
    
    func main() {
    	var pstruct P
    	jsonStr := `{"num":123, "cip":"127.0.0.1","oip":"10.206.85.204",oid":"52852","aid":"6079759","pcid":"75","cid":"76"}`
    	//json.Unmarshal([]byte(jsonStr), &pstruct)
    	sonic.Unmarshal([]byte(jsonStr), &pstruct)
    	fmt.Println(pstruct.Cip) // output: "127.0.0.1"
          fmt.Println(pstruct.Num) // output:123
          fmt.Println(pstruct.Aid) // output: ""
          fmt.Println(pstruct.Oid) // output: ""
    }
    

    if this is normal, how make this print correctly? PS: sonic v1.1.1

    i found out the problem rule: when the type of struct defined is not match with the json, the field and the fields behind it all empty. eg :

    type Test struct {
         Num1 string `json:"num1"`
         Num2 string `json:"num2"`
         Num3 string `json:"num3"`
         Num4 string `json:"num4"`
         Num5 string `json:"num5"`
     }
    testStr := `{"num1":"123","num2":"this is string2", "num3":456, "num4":789, "num5":"this is string"}`
    var t Test
    sonic.Unmarshal([]byte(testStr), &t)
    fmt.Printf("%#v", t)
    //output:  {Num1:"123", Num2:"this is string2", Num3:"", Num4:"", Num5:""}
    
  • ast.Node Get a key that doesn't exist

    ast.Node Get a key that doesn't exist

    http: panic serving 127.0.0.1:44846: runtime error: invalid memory address or nil pointer dereference goroutine 2315 [running]: net/http.(*conn).serve.func1() /usr/local/go/src/net/http/server.go:1802 +0xb9 panic({0xd8a3c0, 0x171dd60}) /usr/local/go/src/runtime/panic.go:1047 +0x266 github.com/bytedance/sonic/ast.(*Node).checkRaw(0xc02713a1b0) /home/zhounan1/gopath/src/github.com/bytedance/sonic/ast/node.go:159 +0x14 github.com/bytedance/sonic/ast.(*Node).String(0x0) /home/zhounan1/gopath/src/github.com/bytedance/sonic/ast/node.go:245 +0x25

    my program just like this:

    list.Get("res").Index(key).Get("item").Get("json_data").Index(cardKey).String()

    When using ast.Node to get a non-existing key, the program will panic. Is there a way to get the null value of the specified type like the get method of simplejson?

  • 生产环境报错,程序重启,求救。

    生产环境报错,程序重启,求救。

    生产环境偶尔报这个错误,recover不到这个异常,导致程序重启。

    traceback: unexpected SPWRITE function github.com/bytedance/sonic/internal/native/avx2.__native_entry__
    fatal error: traceback
    
    runtime stack:
    runtime.throw({0x12ef91c, 0x1de4260})
    	/usr/local/go/src/runtime/panic.go:1198 +0x71
    runtime.gentraceback(0x7fc3972db850, 0x1580, 0xc01949fba0, 0xc0eb5aa4e0, 0x0, 0x0, 0x7fffffff, 0x7fc385ff7460, 0x400000002, 0x0)
    	/usr/local/go/src/runtime/traceback.go:238 +0x19c5
    runtime.addOneOpenDeferFrame.func1()
    	/usr/local/go/src/runtime/panic.go:751 +0x6b
    runtime.systemstack()
    	/usr/local/go/src/runtime/asm_amd64.s:383 +0x49
    
    goroutine 6113840 [running]:
    runtime.systemstack_switch()
    	/usr/local/go/src/runtime/asm_amd64.s:350 fp=0xc000367488 sp=0xc000367480 pc=0x4d08a0
    runtime.addOneOpenDeferFrame(0xc8, 0x1fffffff, 0x47012d)
    	/usr/local/go/src/runtime/panic.go:750 +0x77 fp=0xc0003674c8 sp=0xc000367488 pc=0x49e537
    panic({0x11df0a0, 0x1d84b40})
    	/usr/local/go/src/runtime/panic.go:998 +0x114 fp=0xc000367588 sp=0xc0003674c8 pc=0x49f354
    runtime.panicmem(...)
    	/usr/local/go/src/runtime/panic.go:221
    runtime.sigpanic()
    	/usr/local/go/src/runtime/signal_unix.go:735 +0x327 fp=0xc0003675d8 sp=0xc000367588 pc=0x4b5c47
    github.com/bytedance/sonic/internal/native/avx2.__native_entry__()
    	/data/gopath/pkg/mod/github.com/bytedance/[email protected]/internal/native/avx2/native_amd64.s:1618 +0x1565 fp=0xc0003675e0 sp=0xc0003675d8 pc=0x910925
    created by gateway/handler.selectAppItemsBidResponse
    	/home/ubuntu/remotejenkins/workspace/Gears_Docker_tar2/dsp-gateway/src/handler/dispatcher-handler.go:114 +0x2af
    
  • go test would fail when GO111MODULE=off

    go test would fail when GO111MODULE=off

    prepare env

    export GO111MODULE="on"
    git clone https://github.com/bytedance/sonic.git
    cd sonic
    
    go mod vendor
    

    run go test with GOPATH mode

    GO111MODULE=off go test -v
    

    would fail with the following error

    Begin GC looping...
    === RUN   TestDecodeFloat
    --- PASS: TestDecodeFloat (0.01s)
    === RUN   TestMarshal
    --- FAIL: TestMarshal (0.06s)
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    	panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]
    
    goroutine 9 [running]:
    testing.tRunner.func1.2({0x6a9ae0, 0x8cf6d0})
    	/home/liuqi.victor/opt/go/src/testing/testing.go:1209 +0x24e
    testing.tRunner.func1()
    	/home/liuqi.victor/opt/go/src/testing/testing.go:1212 +0x218
    panic({0x6a9ae0, 0x8cf6d0})
    	/home/liuqi.victor/opt/go/src/runtime/panic.go:1038 +0x215
    github.com/bytedance/sonic/encoder.encodeTypedPointer(0xc0002a6060, 0x6d7d00, 0xc00029e068, 0x7f551ed74528, 0x0)
    	/home/liuqi.victor/go/src/github.com/bytedance/sonic/encoder/primitives.go:75 +0x10d
    github.com/bytedance/sonic/encoder.EncodeInto(0xc0002a6060, {0x6d7d00, 0xc000258d80}, 0x0)
    	/home/liuqi.victor/go/src/github.com/bytedance/sonic/encoder/encoder.go:125 +0xf1
    github.com/bytedance/sonic/encoder.Encode({0x6d7d00, 0xc000258d80}, 0x0)
    	/home/liuqi.victor/go/src/github.com/bytedance/sonic/encoder/encoder.go:99 +0xc5
    github.com/bytedance/sonic.Marshal(...)
    	/home/liuqi.victor/go/src/github.com/bytedance/sonic/sonic.go:36
    github.com/bytedance/sonic.TestMarshal(0xc0000ba4e0)
    	/home/liuqi.victor/go/src/github.com/bytedance/sonic/decode_test.go:1023 +0x3a
    testing.tRunner(0xc0000ba4e0, 0x6f8688)
    	/home/liuqi.victor/opt/go/src/testing/testing.go:1259 +0x102
    created by testing.(*T).Run
    	/home/liuqi.victor/opt/go/src/testing/testing.go:1306 +0x35a
    exit status 2
    FAIL	github.com/bytedance/sonic	0.083s
    

    part go env

    GO111MODULE="on"
    GOARCH="amd64"
    GOVERSION="go1.17.5"
    
  • style: using gofmt to format your code

    style: using gofmt to format your code

    It is weird to see

    import (
        `io/ioutil`
        `testing`
    
        `github.com/stretchr/testify/require`
    )
    

    where use `` instead of ""

    if we want others to participate in, we should follow some basic go-style code style

    just use gofmt -w . in /sonic to format all *.go files, please

  • feat: stop sigprof while calling native

    feat: stop sigprof while calling native

    Backgroud

    • When go received a SIGPROF signal while it is running on sonic's native functions, it will crash duet to missing stack information. At present, sonic doesn't support add stack map for native functions (on plan), thus a work-around way is to set runtime.prof.hz = 0, thus sigprof() handler won't call gentraceback() which cause crash. After we support make stackmap in future, this behavior will be removed

    Benchmark

    • Get
    name                      old time/op    new time/op    delta
    GetOne_Parallel_Sonic-16    3.54µs ±39%    4.85µs ±27%  +36.92%  (p=0.004 n=10+9)
    GetOne_Sonic-16             14.9µs ±20%    12.7µs ±14%  -14.79%  (p=0.001 n=10+9)
    
    name                      old speed      new speed      delta
    GetOne_Parallel_Sonic-16  3.84GB/s ±43%  2.76GB/s ±34%  -28.22%  (p=0.004 n=10+9)
    GetOne_Sonic-16            879MB/s ±18%  1009MB/s ±16%  +14.71%  (p=0.004 n=10+10)
    
    name                      old alloc/op   new alloc/op   delta
    GetOne_Sonic-16              24.0B ± 0%     24.0B ± 0%     ~     (all equal)
    GetOne_Parallel_Sonic-16     24.6B ± 2%     24.6B ± 2%     ~     (p=1.000 n=10+10)
    
    name                      old allocs/op  new allocs/op  delta
    GetOne_Sonic-16               1.00 ± 0%      1.00 ± 0%     ~     (all equal)
    GetOne_Parallel_Sonic-16      1.00 ± 0%      1.00 ± 0%     ~     (all equal)
    
    • flame graph before this PR image

    • flame graph after this PR image

  • fuzz : unmarshal specail float64 filed with string option is different from encoding/json

    fuzz : unmarshal specail float64 filed with string option is different from encoding/json

    code:

    func TestFloat64WithStringOption(t *testing.T) {
    	var f, jf FloatString 
    	var data = []byte("{\"Q\":\"00010\"}")
    	err := sonic.Unmarshal(data, &f)
    	jerr := json.Unmarshal(data, &jf)
    	spew.Dump(err, jerr)
    	spew.Dump(f, jf)
    }
    
    type FloatString struct {
    	Q float64 `json:",string"`
    }
    
    

    output:

    (decoder.SyntaxError) "Syntax error at index 7: invalid char\n\n\t{\"Q\":\"00010\"}\n\t.......^.....\n"
    (interface {}) <nil>
    (issue_test.FloatString) {
     Q: (float64) 0
    }
    (issue_test.FloatString) {
     Q: (float64) 10
    }
    
  • base64x use avx2 simd cause panic

    base64x use avx2 simd cause panic

    use avx2 mode in base64

    const (
        _MODE_JSON = (1 << 3) | (1 << 2) // base64 mode
    )
    
    

    test command:

    go test -run=TestIssue213$  -count 100 -v  > debug.out 
    

    output:

    Begin GC looping...
    === RUN   TestIssue213
    --- PASS: TestIssue213 (0.03s)
    === RUN   TestIssue213
    --- PASS: TestIssue213 (0.00s)
    === RUN   TestIssue213
    fatal error: unexpected signal during runtime execution
    [signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x102410b]
    
    runtime stack:
    runtime.throw({0x13a029a, 0x80c004779fff})
    	/Users/bytedance/goroot/go1.17/src/runtime/panic.go:1198 +0x71
    runtime.sigpanic()
    	/Users/bytedance/goroot/go1.17/src/runtime/signal_unix.go:719 +0x396
    runtime.(*sweepLocked).sweep(0x293305b8, 0x0)
    	/Users/bytedance/goroot/go1.17/src/runtime/mgcsweep.go:436 +0x12b
    runtime.(*mcentral).uncacheSpan(0x104e5c6, 0x70000c064e80)
    	/Users/bytedance/goroot/go1.17/src/runtime/mcentral.go:223 +0
    
  • Compile error on centos system

    Compile error on centos system

    Does anyone develop native code on Centos?

    My ENV: Intel Xeon, Centos 8, Python 3.9.0, clang version 13.0.0, get some compile error with the latest code: (py39_env) [eric@icx250 sonic]$ make mkdir -p internal/native/avx/ echo '// Code generated by Makefile, DO NOT EDIT.' > internal/native/avx/native_amd64.go echo >> internal/native/avx/native_amd64.go sed -e 's/{{PACKAGE}}/avx/g' internal/native/native_amd64.tmpl >> internal/native/avx/native_amd64.go mkdir -p output/avx clang -mno-red-zone -fno-asynchronous-unwind-tables -fno-builtin -fno-exceptions -fno-rtti -fno-stack-protector -nostdlib -O3 -Wall -Werror -Wno-misleading-indentation -Wno-incompatible-pointer-types -msse -mno-sse4 -mavx -mno-avx2 -DUSE_AVX=1 -DUSE_AVX2=0 -S -o output/avx/native.s native/native.c python3 tools/asm2asm/asm2asm.py internal/native/avx/native_amd64.s output/avx/native.s Traceback (most recent call last): File "/home/download/sonic_dir/sonic/tools/asm2asm/asm2asm.py", line 2132, in main() File "/home/download/sonic_dir/sonic/tools/asm2asm/asm2asm.py", line 2067, in main asm.parse(src, proto) File "/home/download/sonic_dir/sonic/tools/asm2asm/asm2asm.py", line 1979, in parse self._parse(src) File "/home/download/sonic_dir/sonic/tools/asm2asm/asm2asm.py", line 1857, in _parse raise SyntaxError('invalid assembly command: ' + cmd.cmd) SyntaxError: invalid assembly command: .text make: *** [Makefile:108: internal/native/avx/native_amd64.s] Error 1

    Is this a config problem, or the asm2asm.py just only support the assembly code compile from Mac OS?

Related tags
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

Sep 14, 2022
JSON diff library for Go based on RFC6902 (JSON Patch)

jsondiff jsondiff is a Go package for computing the diff between two JSON documents as a series of RFC6902 (JSON Patch) operations, which is particula

Dec 4, 2022
Get JSON values quickly - JSON parser for Go
Get JSON values quickly - JSON parser for Go

get json values quickly GJSON is a Go package that provides a fast and simple way to get values from a json document. It has features such as one line

Dec 28, 2022
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

Jun 26, 2022
Json-go - CLI to convert JSON to go and vice versa
Json-go - CLI to convert JSON to go and vice versa

Json To Go Struct CLI Install Go version 1.17 go install github.com/samit22/js

Jul 29, 2022
Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

fastjson - fast JSON parser and validator for Go Features Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks. Parses arb

Jan 5, 2023
Fast and flexible JSON encoder for Go
Fast and flexible JSON encoder for Go

Jettison Jettison is a fast and flexible JSON encoder for the Go programming language, inspired by bet365/jingo, with a richer features set, aiming at

Dec 21, 2022
Fast JSON serializer for golang.

easyjson Package easyjson provides a fast and easy way to marshal/unmarshal Go structs to/from JSON without the use of reflection. In performance test

Jan 4, 2023
Fast Color JSON Marshaller + Pretty Printer for Golang
Fast Color JSON Marshaller + Pretty Printer for Golang

ColorJSON: The Fast Color JSON Marshaller for Go What is this? This package is based heavily on hokaccha/go-prettyjson but has some noticible differen

Dec 19, 2022
A fast json parser for go

rjson rjson is a json parser that relies on Ragel-generated state machines for most parsing. rjson's api is minimal and focussed on efficient parsing.

Sep 26, 2022
Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang

kazaam Description Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang. This functionality provides

Sep 17, 2021
Fast json for go. Lightweight fork of jsoniter.

jx Fast json for go. Lightweight fork of jsoniter. Features Reduced scope (no reflection or encoding/json adapter) Fuzzing, improved test coverage Dra

Dec 27, 2022
library for working amorphous data (as when you decode json into an interface{})

Introduction Decoding json into an interface{} produces an hierarchical arrangement of four data types: float64, string are 'primative types' and form

Aug 1, 2022
A library to query the godoc.org JSON API.

gopkg This repository provides minimal Go package that makes queries against the godoc.org JSON API. Since that site has mostly been subsumed by pkg.g

Dec 2, 2022
Copy of Golang's json library with IsZero feature

json Copy of Golang's json library with IsZero feature from CL13977 Disclaimer It is a package primary used for my own projects, I will keep it up-to-

Oct 9, 2021
Fork of Go's standard library json encoder

A fork of the Go standard library's json encoder Why? https://github.com/golang/go/issues/6213 was proposed in 2013 but was never accepted. Difference

Nov 25, 2021
JSON Unmarshalling Library

JSAWN (JAY-sawn) This is a JSON library to add to the capabilities of the standard 'encoding/json' library. Unmarshalling The first enhancement is to

Feb 16, 2022
Abstract JSON for golang with JSONPath support

Abstract JSON Abstract JSON is a small golang package provides a parser for JSON with support of JSONPath, in case when you are not sure in its struct

Jan 5, 2023
Small utility to create JSON objects
Small utility to create JSON objects

gjo Small utility to create JSON objects. This was inspired by jpmens/jo. Support OS Mac Linux Windows Requirements Go 1.1.14~ Git Installtion Build $

Dec 8, 2022