AsaiYusuke/JSONPath
This Go library is for retrieving a part of JSON according to the JSONPath query syntax.
The core JSONPath syntax on which this library based:
- Stefan GΓΆssner's JSONPath - XPath for JSON
- Christoph Burgmer's json-path-comparison
- JSONPath Internet Draft Development
Note:
For syntax compatibility among other libraries, please check
Table of Contents
Getting started
Install:
go get github.com/AsaiYusuke/jsonpath
Simple example:
package main
import (
"encoding/json"
"fmt"
"github.com/AsaiYusuke/jsonpath"
)
func main() {
jsonPath, srcJSON := `$.key`, `{"key":"value"}`
var src interface{}
json.Unmarshal([]byte(srcJSON), &src)
output, _ := jsonpath.Retrieve(jsonPath, src)
outputJSON, _ := json.Marshal(output)
fmt.Println(string(outputJSON))
// Output:
// ["value"]
}
Basic design
Ease of development
- PEG separated the JSONPath syntax analyzer from functionality itself to simplify the source.
- Equipped with a large number of unit tests to avoid bugs that lead to unexpected results.
Ease of use
- The error specification enables the library user to handle errors correctly.
Compatibility
- Adopted more of the consensus behavior from the Christoph Burgmer's json-path-comparison.
How to use
* Retrieve one-time or repeated
The Retrieve
function returns retrieved result using JSONPath and JSON object:
output, err := jsonpath.Retrieve(jsonPath, src)
The Parse
function returns a parser-function that completed to check JSONPath syntax. By using parser-function, it can repeat to retrieve with the same JSONPath :
jsonPath, err := jsonpath.Parse(jsonPath)
output1, err1 := jsonPath(src1)
output2, err2 := jsonPath(src2)
:
* Error handling
If there is a problem with the execution of APIs, an error type returned. These error types define the corresponding symptom, as listed below:
Retrieve
, Parse
Syntax check errors from Error type | Message format | Symptom | Ex |
---|---|---|---|
ErrorInvalidSyntax |
invalid syntax (position=%d, reason=%s, near=%s) |
The invalid syntax found in the JSONPath. The reason including in this message will tell you more about it. |
|
ErrorInvalidArgument |
invalid argument (argument=%s, error=%s) |
The argument specified in the JSONPath treated as the invalid error in Go syntax. |
|
ErrorFunctionNotFound |
function not found (function=%s) |
The function specified in the JSONPath is not found. |
|
ErrorNotSupported |
not supported (feature=%s, path=%s) |
The unsupported syntaxes specified in the JSONPath. |
|
Retrieve
, parser-functions
Runtime errors from Error type | Message format | Symptom | Ex |
---|---|---|---|
ErrorMemberNotExist |
member did not exist (path=%s) |
The object/array member specified in the JSONPath did not exist in the JSON object. |
|
ErrorTypeUnmatched |
type unmatched (expected=%s, found=%s, path=%s) |
The node type specified in the JSONPath did not exist in the JSON object. |
|
ErrorFunctionFailed |
function failed (function=%s, error=%s) |
The function specified in the JSONPath failed. |
|
The type checking is convenient to recognize which error happened.
:
_,err := jsonpath.Retrieve(jsonPath, srcJSON)
if err != nil {
switch err.(type) {
case jsonpath.ErrorMemberNotExist:
fmt.printf(`retry with other srcJSON: %v`, err)
continue
case jsonpath.ErrorInvalidArgumentFormat:
return nil, fmt.errorf(`specified invalid argument: %v`, err)
}
:
}
* Function syntax
Function enables to format results by using user defined functions. The function syntax comes after the JSONPath.
There are two ways to use function:
Filter function
The filter function applies a user function to each values in the result to get converted.
Aggregate function
The aggregate function converts all values in the result into a single value.
* Accessing JSON
You can get the accessors ( Getters / Setters ) of the input JSON instead of the retrieved values. These accessors can use to update for the input JSON.
This feature can get enabled by giving Config.SetAccessorMode()
.
Note:
It is not possible to use Setter for some results, such as for JSONPath including function syntax.
Also, operations using accessors follow the map/slice manner of Go language. If you use accessors after changing the structure of JSON, you need to pay attention to the behavior. If you don't want to worry about it, get the accessor again every time you change the structure.
Differences
Some behaviors that differ from the consensus exists in this library. For the entire comparisons, please check
These behaviors will change in the future if appropriate ones found.
Character types
The following character types can be available for identifiers in dot-child notation.
Character type | Availabe | Escape |
---|---|---|
* Numbers and alphabets (0-9 A-Z a-z )* Hyphen and underscore ( - _ )* Non-ASCII Unicode characters ( 0x80 - 0x10FFFF ) |
Yes | No |
* Other printable symbols (Space ! " # $ % & ' ( ) * + , . / : ; < = > ? @ [ \ ] ^ ` { | } ~ ) |
Yes | Yes |
* 0x00 - 0x1F , 0x7F ) |
No | - |
The printable symbols except hyphen and underscore can use by escaping them.
JSONPath : $.abc\.def
srcJSON : {"abc.def":1}
Output : [1]
Wildcard in qualifier
The wildcards in qualifier can specify as a union of subscripts.
JSONPath : $[0,1:3,*]
srcJSON : [0,1,2,3,4,5]
Output : [0,1,2,0,1,2,3,4,5]
Regular expression
The regular expression syntax works as a regular expression in Go lang. In particular, you can use "(?i)" to specify the regular expression as the ignore case option.
JSONPath : $[?(@.a=~/(?i)CASE/)]
srcJSON : ["Case","Hello"]
Output : ["Case"]
JSONPaths in the filter-qualifier
JSONPaths that returns value group cannot specify with comparator
or regular expression
. But, existence check
can use these.
JSONPaths that return a value group | example |
---|---|
Recursive descent | @..a |
Multiple identifier | @['a','b'] |
Wildcard identifier | @.* |
Slice qualifier | @[0:1] |
Wildcard qualifier | @[*] |
Union in the qualifier | @[0,1] |
Filter qualifier | @.a[?(@.b)] |
- comparator example (error)
JSONPath : $[?(@..x == "hello world")]
srcJSON : [{"a":1},{"b":{"x":"hello world"}}]
Error : ErrorInvalidSyntax
- regular expression example (error)
JSONPath : $[?(@..x=~/hello/)]
srcJSON : [{"a":1},{"b":{"x":"hello world"}}]
Error : ErrorInvalidSyntax
- existence check example
JSONPath : $[?(@..x)]
srcJSON : [{"a":1},{"b":{"x":"hello world"}}]
Output : [{"b":{"x":"hello world"}}]
Benchmarks
I benchmarked two JSONPaths using several libraries for the Go language. What is being measured is the cost per job for a job that loops a lot after all the prep work done.
There was a performance differences. But if the number of queries is little, there will not be a big difference between any of them.
Also, the results will vary depending on the data entered. So this benchmark is for information only and should be re-measured at every time.
- BenchmarkAsaiYusukeJSONPath
- BenchmarkOhler55Ojg
- BenchmarkBhmjJSONSlice
- BenchmarkPaesslerAGJSONPath
- BenchmarkOliveagleJsonpath
JSONPath for comparison with more libraries
This is the result of a JSONPath that all libraries were able to process. Oliveagle/jsonpath is fastest.
JSONPath : $.store.book[0].price
cpu: Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
BenchmarkAsaiYusukeJSONPath_threeLevelsWithIndex-4 5846179 211.3 ns/op 48 B/op 3 allocs/op
BenchmarkOhler55Ojg_threeLevelsWithIndex-4 1721571 689.4 ns/op 1040 B/op 2 allocs/op
BenchmarkBhmjJSONSlice_threeLevelsWithIndex-4 687261 1844 ns/op 24 B/op 1 allocs/op
BenchmarkPaesslerAGJSONPath_threeLevelsWithIndex-4 1892106 639.2 ns/op 208 B/op 7 allocs/op
BenchmarkOliveagleJsonpath_threeLevelsWithIndex-4 13973798 85.53 ns/op 0 B/op 0 allocs/op
A slightly complex JSONPath
Libraries that can handle complex syntax limited to a few. Among these libraries, my library is fastest.
JSONPath : $..book[?(@.price > $.store.bicycle.price)]
cpu: Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
BenchmarkAsaiYusukeJSONPath_recursiveDescentWithFilter-4 400609 3002 ns/op 464 B/op 20 allocs/op
BenchmarkOhler55Ojg_recursiveDescentWithFilter-4 247669 5094 ns/op 5240 B/op 20 allocs/op
BenchmarkBhmjJSONSlice_recursiveDescentWithFilter-4 59060 20075 ns/op 2936 B/op 57 allocs/op
BenchmarkPaesslerAGJSONPath not supported
BenchmarkOliveagleJsonpath not supported
JSON used for the benchmark measurement:
Show
{ "store": {
"book": [
{ "category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{ "category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
},
{ "category": "fiction",
"author": "Herman Melville",
"title": "Moby Dick",
"isbn": "0-553-21311-3",
"price": 8.99
},
{ "category": "fiction",
"author": "J. R. R. Tolkien",
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8",
"price": 22.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}
Benchmark environment:
Show
Processor : Intel Core i5-6267U 2.90GHz
Memory : 16.0 GB
OS : Windows 10
Go version : go1.16 windows/amd64
Project progress
- Syntax
- Identifier
- identifier in dot notations
- identifier in bracket notations
- wildcard
- multiple-identifier in bracket
- recursive retrieve
- Qualifier
- index
- slice
- wildcard
- Filter
- logical operation
- comparator
- JSONPath retrieve in filter
- script
- Function
- filter
- aggregate
- Refer to the consensus behaviors
- Identifier
- Archtecture
- PEG syntax analyzing
- Error handling
- Function
- Accessing JSON
- Go language manner
- retrieve with the object in interface unmarshal
- retrieve with the json.Number type
- Source code
- Release version
- Unit tests
- syntax tests
- benchmark
- coverage >80%
- Examples
- CI automation
- Documentation
- README
- API doc
- comparison result (local)
- Development status
- determine requirements / functional design
- design-based coding
- testing
- documentation
- Future ToDo
- Refer to the something standard
- Go language affinity
- retrieve with the object in struct unmarshal
- retrieve with the struct tags
- retrieve with the user defined objects