json slicer

Build Status Go Report Card GoDoc

JSON Slice

What is it?

JSON Slice is a Go package which allows to execute fast jsonpath queries without unmarshalling the whole data.

Sometimes you need to get a single value from incoming json using jsonpath, for example to route data accordingly or so. To do that you must unmarshall the whole data into interface{} struct and then apply some jsonpath library to it, only to get just a tiny little value. What a waste of resourses! Well, now there's jsonslice.

Simply call jsonslice.Get on your raw json data to slice out just the part you need. The []byte received can then be unmarshalled into a struct or used as it is.

Getting started

1. install

$ go get github.com/bhmj/jsonslice

2. use it

import "github.com/bhmj/jsonslice"
import "fmt"

func main() {
    var data = []byte(`
    { "sku": [ 
        { "id": 1, "name": "Bicycle", "price": 160, "extras": [ "flashlight", "pump" ] },
        { "id": 2, "name": "Scooter", "price": 280, "extras": [ "helmet", "gloves", "spare wheel" ] }
      ]
    } `)

    a, _ := jsonslice.Get(data, "$.sku[0].price")
    b, _ := jsonslice.Get(data, "$.sku[1].extras.count()")
    c, _ := jsonslice.Get(data, "$.sku[?(@.price > 200)].name")
    d, _ := jsonslice.Get(data, "$.sku[?(@.extras.count() < 3)].name")

    fmt.Println(string(a)) // 160
    fmt.Println(string(b)) // 3
    fmt.Println(string(c)) // ["Scooter"]
    fmt.Println(string(d)) // ["Bicycle"]
}

Run in Go Playground

Package functions

jsonslice.Get(data []byte, jsonpath string) ([]byte, error)

  • get a slice from raw json data specified by jsonpath

Specs

See Stefan Gössner's article for original specs and examples.

Syntax features

  1. Classic dot notation ($.simple_key) is limited to alphanumeric characters. For more complex cases use $['complex key!'] or $.'complex key!'.

  2. A single index reference returns an element, not an array; a slice reference always returns array:

> echo '[{"id":1}, {"id":2}]' | ./jsonslice '$[0].id' 
1
> echo '[{"id":1}, {"id":2}]' | ./jsonslice '$[0:1].id'
[1]
  1. Indexing or slicing on root node is supported (assuming json is an array and not an object):
./jsonslice '$[0].author' sample1.json

Expressions

Overview

  $                   -- root node (can be either object or array)
  .node               -- dot-notated child
  .'some node'        -- dot-notated child (syntax extension)
  ['node']            -- bracket-notated child
  ['foo','bar']       -- bracket-notated children (aggregation)
  [5]                 -- array index
  [-5]                -- negative index means "from the end"
  [1:9]               -- array slice
  [1:9:2]             -- array slice (+step)
  .*  .[*]  .[:]      -- wildcard
  ..key               -- deepscan

Functions

  $.obj.length()      -- number of elements in an array or string length, depending on the obj type
  $.obj.count()       -- same as above
  $.val.size()        -- value size in bytes (as is)

Slices

  $.arr[start:end:step]
  $.arr[start:end]

Selects elements from start (inclusive) to end (exclusive), stepping by step. If step is omitted or zero, then 1 is implied. Out-of-bounds values are reduced to the nearest bound.

If step is positive:

  • empty start treated as the first element inclusive
  • empty end treated as the last element inclusive
  • start should be less then end, otherwise result will be empty

If step is negative:

  • empty start treated as last element inclusive
  • empty end treated as the first element inclusive
  • start should be greater then end, otherwise result will be empty

Filters

  [?(<expression>)]  -- filter expression. Applicable to arrays only
  @                  -- the root of the current element of the array. Used only within a filter.
  @.val              -- a field of the current element of the array.

Filter operators

Operator Description
== Equal to
Use single or double quotes for string expressions.
[?(@.color=='red')] or [?(@.color=="red")]
!= Not equal to
[?(@.author != "Herman Melville")]
> Greater than
[?(@.price > 10)]
>= Greater than or equal to
< Less than
<= Less than or equal to
=~ Match a regexp
[?(@.name =~ /sword.*/i]
!~ or !=~ Don't match a regexp
[?(@.name !~ /sword.*/i]
&& Logical AND
[?(@.price < 10 && @isbn)]
|| Logical OR
[?(@.price > 10 || @.category == 'reference')]

Examples

Assuming sample0.json and sample1.json in the example directory:

cat sample0.json | ./jsonslice '$.store.book[0]'
cat sample0.json | ./jsonslice '$.store.book[0].title'
cat sample0.json | ./jsonslice '$.store.book[0:-1]'
cat sample1.json | ./jsonslice '$[1].author'
cat sample0.json | ./jsonslice '$.store.book[?(@.price > 10)]'
cat sample0.json | ./jsonslice '$.store.book[?(@.price > $.expensive)]'

Much more examples can be found in jsonslice_test.go

Benchmarks (Core i5-7500)

$ go test -bench=. -benchmem -benchtime=4s
goos: linux
goarch: amd64
pkg: github.com/bhmj/jsonslice
++ usually you need to unmarshal the whole JSON to get an object by jsonpath (for reference):
Benchmark_Unmarshal-4                     500000             14712 ns/op            4368 B/op        130 allocs/op
++ and here's a jsonslice.Get:
Benchmark_Jsonslice_Get_Simple-4         2000000              3878 ns/op             128 B/op          4 allocs/op
++ Get() involves parsing a jsonpath, here it is:
Benchmark_JsonSlice_ParsePath-4         10000000               858 ns/op             160 B/op          5 allocs/op
++ in case you aggregate some non-contiguous elements, it may take a bit longer (extra mallocs involved):
Benchmark_Jsonslice_Get_Aggregated-4     1000000              5671 ns/op             417 B/op         13 allocs/op
++ usual unmarshalling a large json:
Benchmark_Unmarshal_10Mb-4                   100          50744817 ns/op             248 B/op          5 allocs/op
++ jsonslicing the same json, target element is near the start:
Benchmark_Jsonslice_Get_10Mb_First-4     3000000              1851 ns/op             128 B/op          4 allocs/op
++ jsonslicing the same json, target element is near the end: still beats Unmarshal
Benchmark_Jsonslice_Get_10Mb_Last-4          200          38286509 ns/op             133 B/op          4 allocs/op
PASS
ok      github.com/bhmj/jsonslice       83.152s

Changelog

1.0.5 (2020-09-22) -- bugfix: $..many.keys used to trigger on many without recursing deeper on keys.

1.0.4 (2020-05-07) -- bugfix: $* path generated panic.

1.0.3 (2019-12-24) -- bugfix: $[0].foo [{"foo":"\\"}] generated "unexpected end of input".

1.0.2 (2019-12-07) -- nested aggregation ($[:].['a','b']) now works as expected. TODO: add option to switch nested aggregation mode at runtime!

1.0.1 (2019-12-01) -- "not equal" regexp operator added (!=~ or !~).

1.0.0 (2019-11-29) -- deepscan operator (..) added, slice with step $[1:9:2] is now supported, syntax extensions added. GetArrayElements() removed.

0.7.6 (2019-09-11) -- bugfix: escaped backslash at the end of a string value.

0.7.5 (2019-05-21) -- Functions count(), size(), length() work in filters.

$.store.bicycle.equipment[?(@.count() == 2)] -> [["light saber", "apparel"]]

0.7.4 (2019-03-01) -- Mallocs reduced (see Benchmarks section).

0.7.3 (2019-02-27) -- GetArrayElements() added.

0.7.2 (2018-12-25) -- bugfix: closing square bracket inside a string value.

0.7.1 (2018-10-16) -- bracket notation is now supported.

$.store.book[:]['price','title'] -> [[8.95,"Sayings of the Century"],[12.99,"Sword of Honour"],[8.99,"Moby Dick"],[22.99,"The Lord of the Rings"]]

0.7.0 (2018-07-23) -- Wildcard key (*) added.

$.store.book[-1].* -> ["fiction","J. R. R. Tolkien","The Lord of the Rings","0-395-19395-8",22.99]
$.store.*[:].price -> [8.95,12.99,8.99,22.99]

0.6.3 (2018-07-16) -- Boolean/null value error fixed.

0.6.2 (2018-07-03) -- More tests added, error handling clarified.

0.6.1 (2018-06-26) -- Nested array indexing is now supported.

$.store.bicycle.equipment[1][0] -> "peg leg"

0.6.0 (2018-06-25) -- Regular expressions added.

$.store.book[?(@.title =~ /(dick)|(lord)/i)].title -> ["Moby Dick","The Lord of the Rings"]

0.5.1 (2018-06-15) -- Logical expressions added.

$.store.book[?(@.price > $.expensive && @.isbn)].title -> ["The Lord of the Rings"]

0.5.0 (2018-06-14) -- Expressions (aka filters) added.

$.store.book[?(@.price > $.expensive)].title -> ["Sword of Honour","The Lord of the Rings"]

0.4.0 (2018-05-16) -- Aggregating sub-queries added.

$.store.book[1:3].author -> ["John","William"]

0.3.0 (2018-05-05) -- MVP.

Roadmap

  • length(), count(), size() functions
  • filters: simple expressions
  • filters: complex expressions (with logical operators)
  • nested arrays support
  • wildcard operator (*)
  • bracket notation for multiple field queries ($['a','b'])
  • deepscan operator (..)
  • syntax extensions: $.'keys with spaces'.price
  • flexible syntax: $[0] works on both [1,2,3] and {"0":"abc"}
  • IN (), NOT IN ()
  • Optionally unmarshal the result
  • Option to select aggregation mode (nested or plain)

Contributing

  1. Fork it!
  2. Create your feature branch: git checkout -b my-new-feature
  3. Commit your changes: git commit -am 'Add some feature'
  4. Push to the branch: git push origin my-new-feature
  5. Submit a pull request :)

Licence

MIT

Author

Michael Gurov aka BHMJ

Owner
Comments
  • deepscan issue

    deepscan issue

    two similar jsonpath queries give different results:

    data:=[]byte({ "apiVersion": "v1", "kind": "Pod", "spec": { "containers": [ { "name": "c1", "image": "busybox", "resources": { "limits": { "cpu": "2" } } }, { "name": "c2", "image": "busybox", "resources": { "limits": { "cpu": "2" } } } ] } })

    jsonslice.Get(data, "$.spec.containers[:]")

    gives the expected result:

    [{ "name": "c1", "image": "busybox", "resources": { "limits": { "cpu": "2" } } },{ "name": "c2", "image": "busybox", "resources": { "limits": { "cpu": "2" } } }]

    BUT:

    jsonslice.Get(data, "$..spec.containers[:]") gives:

    [{ "containers": [ { "name": "c1", "image": "busybox", "resources": { "limits": { "cpu": "2" } } }, { "name": "c2", "image": "busybox", "resources": { "limits": { "cpu": "2" } } } ] }]

  • Clarify Get vs. GetArrayElements

    Clarify Get vs. GetArrayElements

    I'm trying to understand whether calling Get is the right answer for my use case. I'm running $..key against the following and yield "top" only:

    {
        "object": {
            "key": "value",
            "array": [
                {"key": "something"},
                {"key": {"key": "russian dolls"}}
            ]
        },
        "key": "top"
    }
    

    Expected:

    [
      "top",
      "value",
      "something",
      {
        "key": "russian dolls"
      },
      "russian dolls"
    ]
    

    See https://github.com/cburgmer/json-path-comparison/blob/Golang_github.com-bhmj-jsonslice/implementations/Golang_github.com-bhmj-jsonslice/main.go for the code.

  • NOT operator

    NOT operator

    can you support NOT operator? what i actually need is a filter to filter out several results based on the value in the fields using regxep:

    for example: [? NOT(@.name =~ /aaa./i || @.name =~ /bbb./i)]

    the package is really useful.

  • filter not working at object root

    filter not working at object root

    input:{"type":"click",name:"button"} expression: $[?(@.type == "click")] expect output: {"type":"click",name:"button"} actual output: []

    I'm not sure this is a standard jsonpath behavior but java's version jsonpath works as expect.

  • Return value for a single result

    Return value for a single result

    Calling Get with $.store.book[0].author on the following json

    {
    	"store": {
    		"book": [
    			{
    				"category": "fiction",
    				"author": "J. R. R. Tolkien",
    				"title": "The Lord of the Rings",
    				"isbn": "0-395-19395-8",
    				"price": 22.99
    			}
    		]
    	}
    }
    

    returns "J. R. R. Tolkien" rather than [ "J. R. R. Tolkien" ]

    According to jsonpath specification https://goessner.net/articles/JsonPath/index.html#e2

    Please note, that the return value of jsonPath is an array, which is also a valid JSON structure. Also see https://jsonpath.com/ examples

    Moreover the result has " in the string, meaning the resulting string will look like ""J. R. R. Tolkien""...

    Is this by design? am I missing anything? :)

  • Use semantic versioning

    Use semantic versioning

    Hi,

    Thank you for this very useful package.

    It would be great if you use semantic versioning in Git tags, otherwise it will not be possible to import the specific versions with Go Module.

    With semantic versioning, the current version should be v1.0.4 instead of 1.0.4.

    https://golang.org/cmd/go/#hdr-Module_compatibility_and_semantic_versioning https://github.com/golang/go/wiki/Modules#semantic-import-versioning

    Cheers

  • Nil Pointer dereference for an invalid query

    Nil Pointer dereference for an invalid query

    dear @bhmj,

    package main 
    import "github.com/bhmj/jsonslice"
    import "fmt"
    
    func main() {
        var data = []byte(`
        { "sku": [ 
            { "id": 1, "name": "Bicycle", "price": 160, "extras": [ "flashlight", "pump" ] },
            { "id": 2, "name": "Scooter", "price": 280, "extras": [ "helmet", "gloves", "spare wheel" ] }
          ]
        } `)
    
        a, _ := jsonslice.Get(data, "$*")
      
        fmt.Println(string(a)) // 160
    
    }
    

    This program crashes at this line all the times. The behaviour is same with such queries starting with any character from keyterminator. Though the query is invalid, crashing is not expected.

    Thank you.

  • changed behaviour for arrays?

    changed behaviour for arrays?

    jsonpath := "$.a.b[:].['c','d']"
    jsonRaw:=[]byte(`{"a":{"b":[{"c":"cc1","d":"dd1"},{"c":"cc2","d":"dd2"}]}}`)
    result, err := jsonslice.Get(jsonRaw, jsonpath)
    

    the result is: ["cc1","dd1","cc2","dd2"]

    it used to be: [["cc1","dd1"],["cc2","dd2"]]

    which makes more sense

  • can you support more functions?

    can you support more functions?

    For example:

    Function | Description | Output -- | -- | -- min() | Provides the min value of an array of numbers | Double max() | Provides the max value of an array of numbers | Double avg() | Provides the average value of an array of numbers | Double

  • Add abstract and strict comparison

    Add abstract and strict comparison

    Release 1.0.6

    • abstract and strict comparison (== and === behave like in JavaScript)
    • cross-typed string/number/boolean comparison
    • strings are now comparable
  • Issue with getting an object has a `-` in the key

    Issue with getting an object has a `-` in the key

    Blow code will return an empty array, as the key runs-on has a -.

    var data = []byte(`
    	 [
    	    { "runs-on": 1, "name": "Bicycle", "price": 160, "extras": [ "flashlight", "pump" ] },
    	    { "runs-on": 2, "name": "Scooter", "price": 280, "extras": [ "helmet", "gloves", "spare wheel" ] }
    	  ]
    	 `)
    a, _ := jsonslice.Get(data, "$[*].runs-on")
    
  • JSONPath comparison and standardisation

    JSONPath comparison and standardisation

    @cburgmer's JSONPath comparison project is currently discussing some issues relating to a proposed implementation of JSONPath known as "Proposal A". May I encourage you to get involved if you are interested in influencing the direction or, indeed, if you want to make your own proposals.

    Similarly, please join us in slack (invitation) for informal discussions on the comparison project and potential JSONPath standardisation.

  • I want to extend the conditional filter date type as shown in the above example. What interface or method do I need to implement?

    I want to extend the conditional filter date type as shown in the above example. What interface or method do I need to implement?

    json := `{
      "arr": [
        {
          "name": "jack",
          "create_time": "2020-01-01",
          "age": 10
        },
        {
          "name": "wow",
          "create_time": "2020-02-01",
          "age": 11
        }
      ]
    }`
    jsonslice.Get([]byte(json), "$.arr[?(@.create_time > format("yyyy-mm-dd", 2019-12-30))].name")
    

    I want to extend the conditional filter date type as shown in the above example. What interface or method do I need to implement?

  • $[*].bar[*].baz broken in 1.0.2

    $[*].bar[*].baz broken in 1.0.2

    It seems the latest change breaks another query:

    • [ ] $[*].bar[*].baz Input:
      [{"bar": [{"baz": "hello"}]}]
      

      Expected output:

      ["hello"]
      

      Actual output:

      [["hello"]]
      

    If you are interested, you can use the json-path-comparison as a regression suite, all current queries are mapped with current and expected results in https://github.com/cburgmer/json-path-comparison/blob/master/regression_suite/Golang_github.com-bhmj-jsonslice.yaml.

A simple Go package to Query over JSON/YAML/XML/CSV Data
A simple Go package to Query over JSON/YAML/XML/CSV Data

A simple Go package to Query over JSON Data. It provides simple, elegant and fast ODM like API to access, query JSON document Installation Install the

Jan 8, 2023
JSON query expression library in Golang.

jsonql JSON query expression library in Golang. This library enables query against JSON. Currently supported operators are: (precedences from low to h

Dec 31, 2022
Get JSON values quickly - JSON parser for Go
Get JSON values quickly - JSON parser for Go

get json values quickly GJSON is a Go package that provides a fast and simple way to get values from a json document. It has features such as one line

Dec 28, 2022
JSON diff library for Go based on RFC6902 (JSON Patch)

jsondiff jsondiff is a Go package for computing the diff between two JSON documents as a series of RFC6902 (JSON Patch) operations, which is particula

Dec 4, 2022
Fast JSON encoder/decoder compatible with encoding/json for Go
Fast JSON encoder/decoder compatible with encoding/json for Go

Fast JSON encoder/decoder compatible with encoding/json for Go

Jan 6, 2023
go-jsonc provides a way to work with commented json by converting it to plain json.

JSON with comments for GO Decodes a "commented json" to "json". Provided, the input must be a valid jsonc document. Supports io.Reader With this, we c

Apr 6, 2022
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

Jun 26, 2022
Json-match - Command line util for matching values in a JSON input

json-match Match JSON input by specifying key and value > json-match -json '{\"p

Jan 12, 2022
yaml-patch is a version of Evan Phoenix's json-patch, which is an implementation of JSON Patch, directly transposed to YAML

yaml-patch yaml-patch is a version of Evan Phoenix's json-patch, which is an implementation of JavaScript Object Notation (JSON) Patch, directly trans

Jan 15, 2022
Json-go - CLI to convert JSON to go and vice versa
Json-go - CLI to convert JSON to go and vice versa

Json To Go Struct CLI Install Go version 1.17 go install github.com/samit22/js

Jul 29, 2022
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

Sep 14, 2022
RESTful-JSON-API - RESTful-JSON-API using Go

RESTful-JSON-API using Go This basic REST-API principle establishes a one-to-one

Feb 15, 2022
JSON Web Token library

About … a JSON Web Token (JWT) library for the Go programming language. Feature complete Full test coverage Dependency free Key management The API enf

Dec 19, 2022
Safe, simple and fast JSON Web Tokens for Go

jwt JSON Web Token for Go RFC 7519, also see jwt.io for more. The latest version is v3. Rationale There are many JWT libraries, but many of them are h

Jan 4, 2023
This package provides json web token (jwt) middleware for goLang http servers

jwt-auth jwt auth middleware in goLang. If you're interested in using sessions, checkout my sessions library! README Contents: Quickstart Performance

Dec 5, 2022
Golang implementation of JSON Web Tokens (JWT)

jwt-go A go (or 'golang' for search engine friendliness) implementation of JSON Web Tokens NEW VERSION COMING: There have been a lot of improvements s

Jan 6, 2023
JSON or YAML configuration wrapper with convenient access methods.

Config Package config provides convenient access methods to configuration stored as JSON or YAML. This is a fork of the original version. This version

Dec 16, 2022
Library providing routines to merge and validate JSON, YAML and/or TOML files
Library providing routines to merge and validate JSON, YAML and/or TOML files

CONFLATE Library providing routines to merge and validate JSON, YAML, TOML files and/or structs (godoc) Typical use case: Make your application config

Sep 26, 2022
🛠 A configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP
🛠 A configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP

config A small configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP. Example func main() {

Dec 11, 2022