This PR implements the ObjectEach function to iterate over the key-value pairs of a JSON object. Its signature is as follows:

ObjectEach(
  data []byte,
  callback func(key []byte, value []byte, valueType ValueType, offset int) error,
)

ObjectEach will invoke callback for each key-value pair in the JSON object contained in data as follows:

• key will be the unescaped key string of the key-value pair
• value will be the raw bytes composing the value of the key-value pair (as if returned by Get(); quotes are stripped from strings, null is represented as an empty byte slice, and all other values are passed as their literal bytes)
• valueType is the JSON data type of value
• offset is the byte offset just past the end of the value of the key-value pair

If any invocation of callback returns a non-nil error, the iteration terminates and that error is returned from ObjectEach itself.

The name ObjectEach was chosen to mirror the analogous ArrayEach, which iterates over the elements of a JSON array with a callback.

Relevant benchmarks:

$ goapp test -bench "BenchmarkJsonParser.*Struct.*" -benchtime 10s -benchmem
BenchmarkJsonParserEachKeyStructMedium-8     1000000         15307 ns/op         544 B/op         11 allocs/op
BenchmarkJsonParserObjectEachStructMedium-8   500000         23988 ns/op         608 B/op         12 allocs/op
BenchmarkJsonParserEachKeyStructSmall-8     10000000          2248 ns/op         176 B/op          7 allocs/op
BenchmarkJsonParserObjectEachStructSmall-8  10000000          2129 ns/op         240 B/op          8 allocs/op

The decreased performance seen on the Medium test is because ObjectEach double-parses some values. For example, when looking up key person.name.fullName, ObjectEach visits the name key within person, which causes the value {"fullName": ..., ... } to be scanned to find its extent. Then, inside the callback (which receives the byte slice for the value of name), we call GetString to extract the value of fullName within this inner object, which scans that object a second time.

The bottom line is that ObjectEach seems very efficient when scanning a leaf object (i.e., an object whose values are primitives), or when the inner objects will not be accessed (it should be efficient at simply skipping object values), but there is some overhead when values that are objects will be accessed further; in this case, EachKey or simple Gets may be a better choice.

https://github.com/buger/jsonparser/blob/f0ef2b717711d101fbf0856ca3cf58e8199a09cf/parser.go#L126

Escapes can appear in both keys and values in json. The code linked to just compares the raw byte values. See eg https://tools.ietf.org/html/rfc7159#section-8.3

First, thank you for this package!

What do you think about adding a new helper function: GetInt(data []byte, keys ...string) (val int64, offset int, err error)? I think this is a frequent use-case, so this function would be quite useful.

Also to increase the performances of GetNumber and GetInt, you could copy the strconv package from the stdlib into your package and update the signatures of ParseFloat and ParseInt to use []byte instead of string. It would avoid allocating memory and make your package even faster.

I think this is the kind of optimization that is acceptable in a performance-oriented package like yours.

What do you think?

Pardon the unhelpful PR title; wasn't sure what to call this...

Removed the unsafeBytesToString function, instead encapsulating unsafe functionality in a separate set of files (fastbytes*), thus preventing misuse by client code. Included are fast implementations of BytesEqualStr, which takes one string and one []byte argument, and BytesParseFloat and BytesParseInt, which each take a []byte argument. None of these functions let unsafe strings escape, thus improving overall safety.

The main benefit of this refactoring is the ability to have non-unsafe fallback versions of these functions for platforms without the unsafe package (such as Google App Engine). Such implementations are included in fastbytessafe.go.

A somewhat faster implementation of BytesEqualStr is included than was previously used in the codebase. Sadly, string equality checks are a small portion of overall Get*() cost now, so there is little net performance impact. A significantly faster implementation of BytesParseInt is also included, but has little net impact for the same reason.

No noticeable net change in the benchmark, however the code is now cleaner, safer, and more portable.

Note that this patch exports Bytes{EqualStr,ParseInt,ParseFloat}, as they may be of use to client code. This is far safer than the old method of exporting GetUnsafeString, as these new functions are safe.

Benchmark before change:

BenchmarkJsonParserLarge-8    100000         62860 ns/op           0 B/op          0 allocs/op
BenchmarkJsonParserMedium-8   500000         10639 ns/op           0 B/op          0 allocs/op
BenchmarkJsonParserSmall-8  10000000           985 ns/op           0 B/op          0 allocs/op

Benchmark after change:

BenchmarkJsonParserLarge-8    100000         62428 ns/op           0 B/op          0 allocs/op
BenchmarkJsonParserMedium-8  1000000         10779 ns/op           0 B/op          0 allocs/op
BenchmarkJsonParserSmall-8  10000000           996 ns/op           0 B/op          0 allocs/op

Users will often want to convert a []byte returned by Get() or ArrayEach() that represents a primitive value (Null, Boolean, Number, String) into the corresponding Go type (nil, bool, float64, string). This PR provides a convenience method, ParsePrimitiveValue([]byte) (interface{}, error), which performs this service for such users.

Since this is just a convenience method, Get(), etc. are unchanged (except some gofmt whitespace diff in searchKeys...)

Would it be possible to use the parser to get the JSON's Structure?

I know there are existing packages out there, but all that I tried can't keep the original JSON's Structure's order. I.e., the output structure are all sorted by keys, which is both a blessed and cursed at the same time.

Since this parser keeps the original order, it seems to be the only candidate that is able to do it.

Thanks

I want to first thank you, @buger, for your work on this library. Looking up a few JSON key paths in large JSON blobs is a significant bottleneck in a project I'm working on, and your library could give us a big speedup without changing our data format.

Unfortunately, I've discovered an issue in Get(): when searching for a key, Get() may locate that key outside the current JSON object. Here is an example test case that breaks (written using check.v1):

package jsonparser_test

import (
    "github.com/buger-jsonparser"
    . "gopkg.in/check.v1"
    "testing"
)

func (s *JsonParserTests) TestJsonParserSearchBleed(c *C) {
    killer := []byte(`{
      "parentkey": {
        "childkey": {
          "grandchildkey": 123
        },
        "otherchildkey": 123
      }
    }`)

    var jtype int

    _, jtype, _, _ = jsonparser.Get(killer, "childkey")
    c.Assert(jtype, Equals, jsonparser.NotExist) // fails, returns data parentkey.childkey

    _, jtype, _, _ = jsonparser.Get(killer, "parentkey", "childkey", "otherchildkey")
    c.Assert(jtype, Equals, jsonparser.NotExist) // fails, returns data parentkey.otherchildkey
}

// Boilerplate
func Test(t *testing.T) { TestingT(t) }
type JsonParserTests struct{}
var _ = Suite(&JsonParserTests{})

The issue is that Get() uses bytes.Index() to find the next key it's looking for, but only validates it by checking that it is surrounded by double quotes and followed by a colon. In particular, it does not check whether it has crossed an unmatched sequence of braces, which would indicate transitioning into another JSON object level.

I don't have a great suggestion as to how to fix this, sadly. Best of luck.

payload: func main() { testJson := [ s, _ := jsonparser.GetString([]byte(testJson), testJson) fmt.Println(s) }

panic: runtime error: slice bounds out of range [1:0]

goroutine 1 [running]: github.com/buger/jsonparser.searchKeys(0x2c050000, 0x1, 0x1, 0xc0000d7e78, 0x1, 0x1, 0xc00003a000) D:/Go/golibsrc/src/github.com/buger/jsonparser/parser.go:311 +0xfdb github.com/buger/jsonparser.internalGet(0x2c050000, 0x1, 0x1, 0xc0000d7e78, 0x1, 0x1, 0xc0000d7d38, 0x65e120, 0x56afb0, 0xc0000d7dc0, ...) D:/Go/golibsrc/src/github.com/buger/jsonparser/parser.go:891 +0x3a6 github.com/buger/jsonparser.Get(0x2c050000, 0x1, 0x1, 0xc0000d7e78, 0x1, 0x1, 0xc0000d7e87, 0x0, 0xc0000d7e14, 0xc0000d7e87, ...) D:/Go/golibsrc/src/github.com/buger/jsonparser/parser.go:885 +0x90 github.com/buger/jsonparser.GetString(0x2c050000, 0x1, 0x1, 0xc0000d7e78, 0x1, 0x1, 0x9, 0x9, 0x0, 0x0) D:/Go/golibsrc/src/github.com/buger/jsonparser/parser.go:1122 +0x9e

It would be really nice if this package provided a Set(fieldName string, value []byte) or Set(fieldName string, value json.RawMessage) function. Right now we are processing huge amount of big json-line files. Each line has to be unmarshalled as map[string][json.RawMessage](https://golang.org/pkg/encoding/json/#RawMessage), then update just one filed and finally marshal it again. It would be really nice if we could update a value without unmarshalling! What do you think?

Description: Fetch multiple keys at once. Instead of reading whole payload for each key, it does read only 1 time.

Using new API you can get nearly 2x improvement when fetching multiple keys, and the more keys you fetch, the bigger difference.

Example:

paths := [][]string{
    []string{"uuid"},
    []string{"tz"},
    []string{"ua"},
    []string{"st"},
}

jsonparser.EachKey(smallFixture, func(idx int, value []byte, vt jsonparser.ValueType, err error){
    switch idx {
    case 0: // uuid
        // jsonparser.ParseString(value)
    case 1: // tz
        jsonparser.ParseInt(value)
    case 2: // ua
        // jsonparser.ParseString(value)
    case 3: //sa
        jsonparser.ParseInt(value)
    }
}, paths...)

Benchmark before change:

BenchmarkJsonParserLarge      100000         84621 ns/op           0 B/op          0 allocs/op
BenchmarkJsonParserMedium     500000         15803 ns/op           0 B/op          0 allocs/op
BenchmarkJsonParserSmall     5000000          1378 ns/op           0 B/op          0 allocs/op

Added 2 new benchmarks:

BenchmarkJsonParserEachKeyManualMedium   1000000          8312 ns/op           0 B/op          0 allocs/op
BenchmarkJsonParserEachKeyManualSmall   10000000           798 ns/op           0 B/op          0 allocs/op

Added proper JSON escaped string handling in both keys and values (as per RFC 7159). These changes necessarily resulted in some performance degradation, but it's relatively minor (< 10%, it seems).

(Note: this is a refinement of #39)

Benchmark before:

BenchmarkJsonParserLarge-8           64506 ns/op           0 allocs/op
BenchmarkJsonParserMedium-8          10865 ns/op           0 allocs/op
BenchmarkJsonParserSmall-8            1026 ns/op           0 allocs/op

Benchmark after:

BenchmarkJsonParserLarge-8           66558 ns/op           0 allocs/op
BenchmarkJsonParserMedium-8          11958 ns/op           0 allocs/op
BenchmarkJsonParserSmall-8            1061 ns/op           0 allocs/op

Hello, can you tell me the maximum number of byte files that ArrayEach supports? I have a 1GB json file that uses ArrayEach to parse and report an error "offset: 5, err: Value is array, but can't find closing ']' symbol". Thank you.

Description: add a new function DeleteOnOrig which can do delete operation on original slice. It will really help improve performance by reducing one memory alloc/dealloc and one whole copy per one call.

Benchmark before change:

Benchmark after change:

These is no benchmark case because no new code introduced almost.

Hey,

I've been trying to use ArrayEach to parse an array. The callback you can pass takes an error value as its argument and can't return one, meaning that even if you want to stop iterating, you can't. Additionally, the passed error can't be non-nil.

v, t, o, e := Get(data[offset:])

if e != nil {
	return offset, e
}

if o == 0 {
	break
}

if t != NotExist {
	cb(v, t, offset+o-len(v), e)
}

if e != nil {
	break
}

If the error is not nil, we'll run into return offset, e. Since e isn't being reassigned before or after the call to cb, as go does't allow this with type error (value type), the if e != nil { break } is dead code and err will always be nil inside of the callback.

What were the thoughts behind the API design of this function? If I don't understand it, I think comments would be really helpful, both inline and function docs.

Description: Fixes https://github.com/buger/jsonparser/issues/253

Benchmark before change:

Benchmark after change:

For running benchmarks use:

go test -test.benchmem -bench JsonParser ./benchmark/ -benchtime 5s -v
# OR
make bench (runs inside docker)

Sample usage

package main

import (
	"fmt"
	"github.com/buger/jsonparser"
)

func main() {
	data := []byte(`{"menu": {
    "header": "SVG Viewer",
    "items": [
        {"id": "Open"},
        {"id": "OpenNew", "label": "Open New"},
        null,
        {"id": "ZoomIn", "label": "Zoom In"},
        {"id": "ZoomOut", "label": "Zoom Out"},
        {"id": "OriginalView", "label": "Original View"},
        null,
        {"id": "Quality"},
        {"id": "Pause"},
        {"id": "Mute"},
        null,
        {"id": "Find", "label": "Find..."},
        {"id": "FindAgain", "label": "Find Again"},
        {"id": "Copy"},
        {"id": "CopyAgain", "label": "Copy Again"},
        {"id": "CopySVG", "label": "Copy SVG"},
        {"id": "ViewSVG", "label": "View SVG"},
        {"id": "ViewSource", "label": "View Source"},
        {"id": "SaveAs", "label": "Save As"},
        null,
        {"id": "Help"},
        {"id": "About", "label": "About Adobe CVG Viewer..."}
    ]
}}`)
	next, err := jsonparser.ArrayIterator(data, "menu", "items")
	if err != nil {
		fmt.Println("err", err)
	}
	for v, t, o, e := next(); e == nil; {
		fmt.Println(string(v), t, o, e)
		v, t, o, e = next()
		if e != nil {
			fmt.Println("err-loop", e)
		}
	}
}

I am really impressed by this project but I find the current API less helpful.

Hence, proposing a new way to iterate over the array elements or object entries.

Example:

data := []bytes(`{"menu": {
    "header": "SVG Viewer",
    "items": [
        {"id": "Open"},
        {"id": "OpenNew", "label": "Open New"},
        null,
        {"id": "ZoomIn", "label": "Zoom In"},
        {"id": "ZoomOut", "label": "Zoom Out"},
        {"id": "OriginalView", "label": "Original View"},
        null,
        {"id": "Quality"},
        {"id": "Pause"},
        {"id": "Mute"},
        null,
        {"id": "Find", "label": "Find..."},
        {"id": "FindAgain", "label": "Find Again"},
        {"id": "Copy"},
        {"id": "CopyAgain", "label": "Copy Again"},
        {"id": "CopySVG", "label": "Copy SVG"},
        {"id": "ViewSVG", "label": "View SVG"},
        {"id": "ViewSource", "label": "View Source"},
        {"id": "SaveAs", "label": "Save As"},
        null,
        {"id": "Help"},
        {"id": "About", "label": "About Adobe CVG Viewer..."}
    ]
}}`)

// Either support object notation
iterator, err := jsonparser.ArrayIterator(data, "menu", "items")
for val, err = iterator.next(); err == nil && iterator.hasNext(); val, err = iterator.next() {
    // iterator.hasNext() can be replaced by err = errors.New("Done")
    // do anything or break as required.
  }

// OR support closure notation
next(), hasNext, err = jsonparser.ArrayIterator(data, "menu", "items")
for val, hasNext, err = next(); err == nil && hasNext; val, hasNext, err = next() {
  // hasNext can be dropped as err = errors.New("Done") can convey same information.
  // do anything or break as required.
}

Similar syntax can be used for iterating object entries (key/values), or other functions like EachKey which ask for a callback to iterate over values.

This will prevent the callback hell and also make the code more efficient & readable since we need not iterate over all entries to find one.

PS. I can also contribute and create a PR once it is approved.

EachKey gets an array of strings as a parameter, but it would be quite better for me to iterate over map data structure. Could you please make a function with paths specified this like?

myGreatPaths := map[string][]string{
	"event": []string{"event"},
	"user_id": []string{"arg", "uid"},
	...
}