ffjson: faster JSON for Go
ffjson generates static MarshalJSON and UnmarshalJSON functions for structures in Go. The generated functions reduce the reliance upon runtime reflection to do serialization and are generally 2 to 3 times faster. In cases where ffjson doesn't understand a Type involved, it falls back to encoding/json, meaning it is a safe drop in replacement. By using ffjson your JSON serialization just gets faster with no additional code changes.
When you change your struct, you will need to run ffjson again (or make it part of your build tools).
Blog Posts
- 2014-03-31: First Release and Background
Getting Started
If myfile.go contains the struct types you would like to be faster, and assuming GOPATH is set to a reasonable value for an existing project (meaning that in this particular example if myfile.go is in the myproject directory, the project should be under $GOPATH/src/myproject), you can just run:
go get -u github.com/pquerna/ffjson
ffjson myfile.go
git add myfile_ffjson.go
Performance Status:
MarshalJSONis 2x to 3x faster thanencoding/json.UnmarshalJSONis 2x to 3x faster thanencoding/json.
Features
- Unmarshal Support: Since v0.9,
ffjsonsupports Unmarshaling of structures. - Drop in Replacement: Because
ffjsonimplements the interfaces already defined byencoding/jsonthe performance enhancements are transparent to users of your structures. - Supports all types:
ffjsonhas native support for most of Go's types -- for any type it doesn't support with fast paths, it falls back to usingencoding/json. This means all structures should work out of the box. If they don't, open a issue! - ffjson: skip: If you have a structure you want
ffjsonto ignore, addffjson: skipto the doc string for this structure. - Extensive Tests:
ffjsoncontains an extensive test suite including fuzz'ing against the JSON parser.
Using ffjson
ffjson generates code based upon existing struct types. For example, ffjson foo.go will by default create a new file foo_ffjson.go that contains serialization functions for all structs found in foo.go.
Usage of ffjson:
ffjson [options] [input_file]
ffjson generates Go code for optimized JSON serialization.
-go-cmd="": Path to go command; Useful for `goapp` support.
-import-name="": Override import name in case it cannot be detected.
-nodecoder: Do not generate decoder functions
-noencoder: Do not generate encoder functions
-w="": Write generate code to this path instead of ${input}_ffjson.go.
Your code must be in a compilable state for ffjson to work. If you code doesn't compile ffjson will most likely exit with an error.
Disabling code generation for structs
You might not want all your structs to have JSON code generated. To completely disable generation for a struct, add ffjson: skip to the struct comment. For example:
// ffjson: skip
type Foo struct {
Bar string
}
You can also choose not to have either the decoder or encoder generated by including ffjson: nodecoder or ffjson: noencoder in your comment. For instance, this will only generate the encoder (marshal) part for this struct:
// ffjson: nodecoder
type Foo struct {
Bar string
}
You can also disable encoders/decoders entirely for a file by using the -noencoder/-nodecoder commandline flags.
Using ffjson with go generate
ffjson is a great fit with go generate. It allows you to specify the ffjson command inside your individual go files and run them all at once. This way you don't have to maintain a separate build file with the files you need to generate.
Add this comment anywhere inside your go files:
//go:generate ffjson $GOFILE
To re-generate ffjson for all files with the tag in a folder, simply execute:
go generate
To generate for the current package and all sub-packages, use:
go generate ./...
This is most of what you need to know about go generate, but you can sese more about go generate on the golang blog.
Should I include ffjson files in VCS?
That question is really up to you. If you don't, you will have a more complex build process. If you do, you have to keep the generated files updated if you change the content of your structs.
That said, ffjson operates deterministically, so it will generate the same code every time it run, so unless your code changes, the generated content should not change. Note however that this is only true if you are using the same ffjson version, so if you have several people working on a project, you might need to synchronize your ffjson version.
Performance pitfalls
ffjson has a few cases where it will fall back to using the runtime encoder/decoder. Notable cases are:
- Interface struct members. Since it isn't possible to know the type of these types before runtime, ffjson has to use the reflect based coder.
- Structs with custom marshal/unmarshal.
- Map with a complex value. Simple types like
map[string]intis fine though. - Inline struct definitions
type A struct{B struct{ X int} }are handled by the encoder, but currently has fallback in the decoder. - Slices of slices / slices of maps are currently falling back when generating the decoder.
Reducing Garbage Collection
ffjson already does a lot to help garbage generation. However whenever you go through the json.Marshal you get a new byte slice back. On very high throughput servers this can lead to increased GC pressure.
Tip 1: Use ffjson.Marshal() / ffjson.Unmarshal()
This is probably the easiest optimization for you. Instead of going through encoding/json, you can call ffjson. This will disable the checks that encoding/json does to the json when it receives it from struct functions.
import "github.com/pquerna/ffjson/ffjson"
// BEFORE:
buf, err := json.Marshal(&item)
// AFTER:
buf, err := ffjson.Marshal(&item)
This simple change is likely to double the speed of your encoding/decoding.
[![GoDoc][1]][2] [1]: https://godoc.org/github.com/pquerna/ffjson/ffjson?status.svg [2]: https://godoc.org/github.com/pquerna/ffjson/ffjson#Marshal
Tip 2: Pooling the buffer
On servers where you have a lot of concurrent encoding going on, you can hand back the byte buffer you get from json.Marshal once you are done using it. An example could look like this:
import "github.com/pquerna/ffjson/ffjson"
func Encode(item interface{}, out io.Writer) {
// Encode
buf, err := ffjson.Marshal(&item)
// Write the buffer
_,_ = out.Write(buf)
// We are now no longer need the buffer so we pool it.
ffjson.Pool(buf)
}
Note that the buffers you put back in the pool can still be reclaimed by the garbage collector, so you wont risk your program building up a big memory use by pooling the buffers.
[![GoDoc][1]][2] [1]: https://godoc.org/github.com/pquerna/ffjson/ffjson?status.svg [2]: https://godoc.org/github.com/pquerna/ffjson/ffjson#Pool
Tip 3: Creating an Encoder
There might be cases where you need to encode many objects at once. This could be a server backing up, writing a lot of entries to files, etc.
To do this, there is an interface similar to encoding/json, that allow you to create a re-usable encoder. Here is an example where we want to encode an array of the Item type, with a comma between entries:
import "github.com/pquerna/ffjson/ffjson"
func EncodeItems(items []Item, out io.Writer) {
// We create an encoder.
enc := ffjson.NewEncoder(out)
for i, item := range items {
// Encode into the buffer
err := enc.Encode(&item)
// If err is nil, the content is written to out, so we can write to it as well.
if i != len(items) -1 {
_,_ = out.Write([]byte{','})
}
}
}
Documentation: [![GoDoc][1]][2] [1]: https://godoc.org/github.com/pquerna/ffjson/ffjson?status.svg [2]: https://godoc.org/github.com/pquerna/ffjson/ffjson#Encoder
Tip 4: Avoid interfaces
We don't want to dictate how you structure your data, but having interfaces in your code will make ffjson use the golang encoder for these. When ffjson has to do this, it may even become slower than using json.Marshal directly.
To see where that happens, search the generated _ffjson.go file for the text Falling back, which will indicate where ffjson is unable to generate code for your data structure.
Tip 5: ffjson all the things!
You should not only create ffjson code for your main struct, but also any structs that is included/used in your json code.
So if your struct looks like this:
type Foo struct {
V Bar
}
You should also make sure that code is generated for Bar if it is placed in another file. Also note that currently it requires you to do this in order, since generating code for Foo will check if code for Bar exists. This is only an issue if Foo and Bar are placed in different files. We are currently working on allowing simultaneous generation of an entire package.
Improvements, bugs, adding features, and taking ffjson new directions!
Please open issues in Github for ideas, bugs, and general thoughts. Pull requests are of course preferred :)
Similar projects
- go-codec. Very good project, that also allows streaming en/decoding, but requires you to call the library to use.
- megajson. This has limited support, and development seems to have almost stopped at the time of writing.
Credits
ffjson has recieved significant contributions from:
License
ffjson is licensed under the Apache License, Version 2.0
