A simple Go package to Query over JSON/YAML/XML/CSV Data

gojsonq-logo

Build Status Project status Go Report Card Coverage Status GoDoc License

A simple Go package to Query over JSON Data. It provides simple, elegant and fast ODM like API to access, query JSON document

Installation

Install the package using

$ go get github.com/thedevsaddam/gojsonq/v2

Usage

To use the package import it in your *.go code

import "github.com/thedevsaddam/gojsonq/v2"

Let's see a quick example:

See in playground

package main

import gojsonq "github.com/thedevsaddam/gojsonq/v2"

func main() {
	const json = `{"name":{"first":"Tom","last":"Hanks"},"age":61}`
	name := gojsonq.New().FromString(json).Find("name.first")
	println(name.(string)) // Tom
}

Another example:

See in playground

package main

import (
	"fmt"

	gojsonq "github.com/thedevsaddam/gojsonq/v2"
)

func main() {
	const json = `{"city":"dhaka","type":"weekly","temperatures":[30,39.9,35.4,33.5,31.6,33.2,30.7]}`
	avg := gojsonq.New().FromString(json).From("temperatures").Avg()
	fmt.Printf("Average temperature: %.2f", avg) // 33.471428571428575
}

You can query your document using the various query methods such as Find, First, Nth, Pluck, Where, OrWhere, WhereIn, WhereStartsWith, WhereEndsWith, WhereContains, Sort, GroupBy, SortBy and so on. Also you can aggregate data after query using Avg, Count, Max, Min, Sum etc.

Find more query API in Wiki page

Bugs and Issues

If you encounter any bugs or issues, feel free to open an issue at github.

Also, you can shoot me an email to mailto:[email protected] for hugs or bugs.

Credit

Special thanks to Nahid Bin Azhar for the inspiration and guidance for the package. Thanks to Ahmed Shamim Hasan Shaon for his support from the very beginning.

Contributors

Contribution

If you are interested to make the package better please send pull requests or create an issue so that others can fix. Read the contribution guide here

Special Thanks

License

The gojsonq is an open-source software licensed under the MIT License.

Owner
Saddam H
Sr. Software Engineer @ Pathao Inc | Open Source Enthusiast | Love to write elegant code | Golang | Distributed Systems
Saddam H
Comments
  • add handling of objects to processQuery

    add handling of objects to processQuery

    Change to enable queries on standalone objects. This makes queries like this possible:

    j := gojsonq.New().JSONString(`{"name":{"first":"Tom","last":"Hanks"},"age":61}`)
    ct := j.Where("age", ">=", 60).Where("name.last", "eq", "Hanks").Count() // 1
    
  • NOT an issue, more a touch of help. Using parameters with gojsonq package

    NOT an issue, more a touch of help. Using parameters with gojsonq package

    I am trying to build a server that can fetch certain json entries based on a parameter. So for example if a user enters localhost:8080/store/1 It returns the json entry with the ID 1.

    The code i currently have works if i hardcode the id field into the query, but trying to put in parameters i have so far had no luck. Trying this:

    func getGame(w http.ResponseWriter, r *http.Request) {
    	params := mux.Vars(r)
    	jq := gojsonq.New().File("./data/games.json")
    	res := jq.From("games").Where("id", "=", params["id"]).Get()
    	fmt.Println(res)
    }
    

    Suggestions please, before i throw something out the window.

  • Multiple nested array support

    Multiple nested array support

    Is it possible to traverse multiple nested arrays in this library without knowing the array indexes? I have a scenario that works well with where clauses on the first array, but I haven't had any success with a second array below that.

    This is a successful query using Find (but assumes I know the index to each array):

    Find("data.items.[1].nestedA.[3].nestedB.value")

    I've tried many variations of: From("data.items").Where("itemValue", "=", "asdf").From("nestedA").Where("nestedBOtherValue", "eq", "fdsa").Find("nestedB.value") without any luck.

  • Add more error info on As() function

    Add more error info on As() function

    issue

    When using :

    tempTags = []Tag{} // Structure
    jsonResult.As(&tempTags)
    

    This error is displayed an error from reflect library:

    panic: reflect: call of reflect.Value.Call on zero Value [recovered]
            panic: reflect: call of reflect.Value.Call on zero Value
    

    Expected

    Should display a more specific error that give us more clues in order to understand what is going on

    Panic: the method '*[]repository.Tag' must be part of 'map[*[]bool:BoolSlice *[]float32:Float32Slice *[]float64:Float64Slice *[]int:IntSlice *[]int16:Int16Slice *[]int32:Int32Slice *[]int8:Int8Slice *[]string:StringSlice *[]time.Duration:DurationSlice *[]uint:UintSlice *[]uint16:Uint16Slice *[]uint32:Uint32Slice *[]uint8:Uint8Slice *bool:Bool *float32:Float32 *float64:Float64 *int:Int *int16:Int16 *int32:Int32 *int8:Int8 *string:String *time.Duration:Duration *uint:Uint *uint16:Uint16 *uint32:Uint32 *uint8:Uint8]' [recovered]
            panic: the method '*[]repository.Tag' must be part of 'map[*[]bool:BoolSlice *[]float32:Float32Slice *[]float64:Float64Slice *[]int:IntSlice *[]int16:Int16Slice *[]int32:Int32Slice *[]int8:Int8Slice *[]string:StringSlice *[]time.Duration:DurationSlice *[]uint:UintSlice *[]uint16:Uint16Slice *[]uint32:Uint32Slice *[]uint8:Uint8Slice *bool:Bool *float32:Float32 *float64:Float64 *int:Int *int16:Int16 *int32:Int32 *int8:Int8 *string:String *time.Duration:Duration *uint:Uint *uint16:Uint16 *uint32:Uint32 *uint8:Uint8]'
    
  • Query json for array size.

    Query json for array size.

    Hello,

    This is an "info request".

    I have the following json and I would like to check if the input contains a father (fid) with two children.

    {
      "data": [
        {
          "fid": 10,
          "children": [
            {
              "cid": 123
            },
            {
              "cid": 124
            }
           
          ]
        }
      ]
    }
    

    The code so far is

    package main
    
    import (
    	"fmt"
    
    	"github.com/thedevsaddam/gojsonq"
    )
    
    const json = `{
    	"data": [
    	  {
    		"fid": 10,
    		"children": [
    		  {
    			"cid": 123
    		  },
    		  {
    			"cid": 124
    		  }
    		 
    		]
    	  }
    	]
      }`
    
    func main() {
    
    	q := gojsonq.New().JSONString(json).From("data").
    		Where("fid", "=", 10)
    	fmt.Println(q.Get())
    }
    
    

    How can I modify the source code to decide if fid=10 AND children.size() = 2?

    In principle how can I use the "array size" to query the json file/input?

    Thanks

  • Map[string]interface{} root

    Map[string]interface{} root

    Doesn't seem to work when you don't have nicely sliceable structures.

    IE: https://www.iinet.net.au/status/event/events.json

    I tried Where("DRW", "=", "Darwin") but kept getting nothing back

  • FromInterface custom decoder

    FromInterface custom decoder

    Hi there, Is there any way to load JSON document from map[string]interface{}? Simplest use case is querying MongoDB, which returns result documents in that format(without using schema struct). I am willing to make a PR if little help is supplied. Thanks

  • Go-critic warnings

    Go-critic warnings

    go-critic linter found some issues , here corresponding log:

    check-package: $GOPATH\src\github.com\thedevsaddam\gojsonq\helper.go:45:2: typeS witchVar: case 0 can benefit from type switch with assignment check-package: $GOPATH\src\github.com\thedevsaddam\gojsonq\helper.go:45:2: typeS witchVar: case 1 can benefit from type switch with assignment check-package: $GOPATH\src\github.com\thedevsaddam\gojsonq\helper.go:45:2: typeS witchVar: case 2 can benefit from type switch with assignment check-package: $GOPATH\src\github.com\thedevsaddam\gojsonq\helper.go:45:2: typeS witchVar: case 3 can benefit from type switch with assignment check-package: $GOPATH\src\github.com\thedevsaddam\gojsonq\helper.go:45:2: typeS witchVar: case 4 can benefit from type switch with assignment check-package: $GOPATH\src\github.com\thedevsaddam\gojsonq\helper.go:45:2: typeS witchVar: case 5 can benefit from type switch with assignment check-package: $GOPATH\src\github.com\thedevsaddam\gojsonq\helper.go:45:2: typeS witchVar: case 6 can benefit from type switch with assignment check-package: $GOPATH\src\github.com\thedevsaddam\gojsonq\jsonq.go:271:3: range ValCopy: each iteration copies 48 bytes (consider pointers or indexing) check-package: $GOPATH\src\github.com\thedevsaddam\gojsonq\jsonq.go:45:18: typeU nparen: could simplify to [][]query

    In #18 I propose fixing for them

    Why replacerange with for loop? You must create an issue regarding this refactor before sending PR.

    Thanks, good point. The way that I made in my PR not very accurate and should be replaced into this one as written in description to rangeValCopy.

  • Added documentation and examples

    Added documentation and examples

    I saw a post on r/golang probably yours. And I wondered why you did not add the examples in the linked article to your project, so I did add some and of your README-file. These will be shown on https://godoc.org/github.com/thedevsaddam/gojsonq the godoc page.

    You have to edit the import-path of the examples_test.go file to your repo as I plan to remove my fork as soon as the changes are merged.

  • Why interfaces?

    Why interfaces?

    Thanks for sharing this project. One question. Why do you do this:

    // strStrictContains checks if x contains y
    // This is case sensitive search
    func strStrictContains(x, y interface{}) (bool, error) {
    	xv, okX := x.(string)
    	if !okX {
    		return false, fmt.Errorf("%v must be string", x)
    	}
    	yv, okY := y.(string)
    	if !okY {
    		return false, fmt.Errorf("%v must be string", y)
    	}
    	return strings.Contains(xv, yv), nil
    }
    

    ...and not this:

    // strStrictContains checks if x contains y
    // This is case sensitive search
    func strStrictContains(x, y string) (bool) {
    	return strings.Contains(xv, yv)
    }
    

    ...or in this case, simply strings.Contains(xv, yv) without a wrapper.

    In this example, you are only allowing string inputs anyway, so why not use strongly typed strings as input params instead of interface{}?

  • Support for unselect

    Support for unselect

    I have an array of JSON objects which all contain a few common keys and then a whole bunch of keys that are different. I want to select a subset of this array that drops a single common key.

    The only way to do that right now is to use a select or only with all the possible keys the object can contain minus the one I want to drop. It would be nice to support something like "unselect" or "drop" where I can drop a specific key from the result while keeping the rest.

  • Getting same results for both `eq` and `neq` clauses

    Getting same results for both `eq` and `neq` clauses

    data:

    {
      "appStatus": {
        "displayName": "JobServer",
        "instanceStatus": {
          "autoRestartSuspended": false,
          "autoRestartSuspendedForApp": false,
          "autoRestartableApp": false,
          "instance": {
            "active": true,
            "applicationDisplayName": "JobServer",
            "applicationId": 42,
            "applicationName": "JobServer",
            "custom": false,
            "id": 10,
            "name": 1,
            "serverName": "xxx"
          },
          "lastStartTime": "2022-11-06 11:53:52",
          "lastStatusChangeTime": "2022-12-14 16:37:31",
          "lastStatusEvent": "Stop",
          "lastStopTime": "2022-12-14 16:37:31",
          "status": "STOPPED"
        },
        "name": "JobServer",
        "numberOfInstancesDown": 1,
        "numberOfInstancesRunning": 0,
        "outageCountInLast24Hrs": 1,
        "pecentageUpInstance": 0,
        "status": "Red",
        "totalNumberOfInstances": 1
      }
    }
    

    code:

    notStopped := jq.New().File("out.json").
    	From("appStatus.instanceStatus").
    	Where("status", "neq", "STOPPED")
    
    log.Println(notStoppped)
    log.Println(notStopped.Get())
    

    output:

    Content: {
      "appStatus": {
        "displayName": "JobServer",
        "instanceStatus": {
          "autoRestartSuspended": false,
          "autoRestartSuspendedForApp": false,
          "autoRestartableApp": false,
          "instance": {
            "active": true,
            "applicationDisplayName": "JobServer",
            "applicationId": 42,
            "applicationName": "JobServer",
            "custom": false,
            "id": 10,
            "name": 1,
            "serverName": "xxx"
          },
          "lastStartTime": "2022-11-06 11:53:52",
          "lastStatusChangeTime": "2022-12-14 16:37:31",
          "lastStatusEvent": "Stop",
          "lastStopTime": "2022-12-14 16:37:31",
          "status": "STOPPED"
        },
        "name": "JobServer",
        "numberOfInstancesDown": 1,
        "numberOfInstancesRunning": 0,
        "outageCountInLast24Hrs": 1,
        "pecentageUpInstance": 0,
        "status": "Red",
        "totalNumberOfInstances": 1
      }
    }
    
    Queries:[[{status eq STOPPED}]]
    
    2022/12/14 20:48:31 map[autoRestartSuspended:false autoRestartSuspendedForApp:false autoRestartableApp:false instance:map[active:true applicationDisplayName:JobServer applicationId:42 applicationName:JobServer custom:false id:10 name:1 serverName:xxx] lastStartTime:2022-11-06 11:53:52 lastStatusChangeTime:2022-12-14 16:37:31 lastStatusEvent:Stop lastStopTime:2022-12-14 16:37:31 status:STOPPED]
    

    Changing where to eq produces exactly same output. I would think that neq should return an empty map instead?

    Another test with neq:

    notStopped := jq.New().File("out.json").
    	From("appStatus").
    	From("instanceStatus.instance").
    	Where("name", "!=", 1)
    

    Returns:

    map[active:true applicationDisplayName:JobServer applicationId:42 applicationName:JobServer custom:false id:10 name:1 serverName:xxx]
    

    So it would appear that neq and != do not work as expected?

    Any help in getting this working? Am I doing smth wrong?

    PS: is this lib maintained? Seems like not much activity going on lately.

  • Add support for byteArray, default values and Object Processing along with JSON Array

    Add support for byteArray, default values and Object Processing along with JSON Array

    Added support for:

    1. Reading JSON byteArray - Decode byte array in to root content
    2. Default values - Support to provide default value for any given non-existing property
    3. Object Processing along with JSON Array - Support to have filter & projection on a single JSON object along with the JSON Array

    https://github.com/thedevsaddam/gojsonq/issues/99

  • Add support for byteArray, default values and Object Processing along with JSON Array

    Add support for byteArray, default values and Object Processing along with JSON Array

    Can we add support for the

    1. Reading JSON byteArray - Decode byte array in to root content
    2. Default values - Support to provide default value for any given non-existing property
    3. Object Processing along with JSON Array - Support to have filter & projection on a single JSON object along with the JSON Array
  • FAIL: TestJSONQ_SortBy_float_ascending_order

    FAIL: TestJSONQ_SortBy_float_ascending_order

    a unit test is failing when compiling with golang1.19

     % go version
    go version go1.19 darwin/amd64
    
    % make test
    go test -timeout 60s  ./...
    --- FAIL: TestJSONQ_SortBy_float_ascending_order (0.00s)
        jsonq_testdata_test.go:134: Tag: sorting array of object by its key (price-float64) in ascending desc
            Expected: [{"id":null,"name":"HP core i3 SSD","price":850},{"id":4,"name":"Fujitsu","price":850},{"id":5,"key":2300,"name":"HP core i5","price":850},{"id":6,"name":"HP core i7","price":950},{"id":3,"name":"Sony VAIO","price":1200},{"id":1,"name":"MacBook Pro 13 inch retina","price":1350},{"id":2,"name":"MacBook Pro 15 inch retina","price":1700}]
            Got: [{"id":4,"name":"Fujitsu","price":850},{"id":5,"key":2300,"name":"HP core i5","price":850},{"id":null,"name":"HP core i3 SSD","price":850},{"id":6,"name":"HP core i7","price":950},{"id":3,"name":"Sony VAIO","price":1200},{"id":1,"name":"MacBook Pro 13 inch retina","price":1350},{"id":2,"name":"MacBook Pro 15 inch retina","price":1700}]
    FAIL
    FAIL	github.com/thedevsaddam/gojsonq/v2	0.353s
    FAIL
    
  • Unable to sort multiple fields with OrderBy

    Unable to sort multiple fields with OrderBy

    Supposed that we have list of objects which have internalScore and externalScore. OrderBy can only sort one field only, either those two. If there are 2 records having the same internalScore, when it's already sorted by internalScore, then externalScore cannot be sorted. So, at the moment OrderBy cannot do this.

    I wish it allowed us to call OrderBy multiple times which adds up instead of overriding it.

    It's similar to SQL functionality:

    SELECT * FROM tbl_Employee ORDER BY lastName ASC, Salary DESC
    
JSON query expression library in Golang.

jsonql JSON query expression library in Golang. This library enables query against JSON. Currently supported operators are: (precedences from low to h

Dec 31, 2022
Simple filter query language parser so that you can build SQL, Elasticsearch, etc. queries safely from user input.

fexpr fexpr is a filter query language parser that generates extremely easy to work with AST structure so that you can create safely SQL, Elasticsearc

Dec 29, 2022
Resource Query Language for REST
Resource Query Language for REST

RQL is a resource query language for REST. It provides a simple and light-weight API for adding dynamic querying capabilities to web-applications that

Jan 2, 2023
Query Parser for REST

Query Parser for REST Query Parser is a library for easy building dynamic SQL queries to Database. It provides a simple API for web-applications which

Dec 24, 2022
GSQL is a structured query language code builder for golang.

GSQL is a structured query language code builder for golang.

Dec 12, 2022
json slicer

JSON Slice What is it? JSON Slice is a Go package which allows to execute fast jsonpath queries without unmarshalling the whole data. Sometimes you ne

Dec 18, 2022
This app is an attempt towards using go lang with graphql data fetch in react front end.

go_movies _A React js + GraphQL supported with backend in GoLang. This app is an attempt towards using go lang with graphql data fetch in react front

Dec 7, 2021
A simple Go, GraphQL, and PostgreSQL starter template

Simple Go/GraphQL/PostgreSQL template Purpose Have a good starting point for any project that needs a graphql, go, and postgres backend. It's a very l

Jan 8, 2022
A simple 2D demo in go using embiten game engine
A simple 2D demo in go using embiten game engine

a little demo in go using the ebiten game engine vaguely follows my memory of ho

Dec 29, 2021
A simple (yet effective) GraphQL to HTTP / REST router

ReGraphQL A simple (yet effective) GraphQL to REST / HTTP router. ReGraphQL helps you expose your GraphQL queries / mutations as REST / HTTP endpoints

Dec 12, 2022
A simple Go package to Query over JSON/YAML/XML/CSV Data
A simple Go package to Query over JSON/YAML/XML/CSV Data

A simple Go package to Query over JSON Data. It provides simple, elegant and fast ODM like API to access, query JSON document Installation Install the

Dec 27, 2022
Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.
Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.

dasel Dasel (short for data-selector) allows you to query and modify data structures using selector strings. Comparable to jq / yq, but supports JSON,

Jan 2, 2023
Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.
Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.

dasel Dasel (short for data-selector) allows you to query and modify data structures using selector strings. Comparable to jq / yq, but supports JSON,

Jan 2, 2023
converts text-formats from one to another, it is very useful if you want to re-format a json file to yaml, toml to yaml, csv to yaml, ... etc

re-txt reformates a text file from a structure to another, i.e: convert from json to yaml, toml to json, ... etc Supported Source Formats json yaml hc

Sep 23, 2022
Dasel - Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool.
Dasel - Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool.

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

Jan 1, 2023
Prometheus Common Data Exporter can parse JSON, XML, yaml or other format data from various sources (such as HTTP response message, local file, TCP response message and UDP response message) into Prometheus metric data.
Prometheus Common Data Exporter can parse JSON, XML, yaml or other format data from various sources (such as HTTP response message, local file, TCP response message and UDP response message) into Prometheus metric data.

Prometheus Common Data Exporter Prometheus Common Data Exporter 用于将多种来源(如http响应报文、本地文件、TCP响应报文、UDP响应报文)的Json、xml、yaml或其它格式的数据,解析为Prometheus metric数据。

May 18, 2022
Universal JSON, BSON, YAML, CSV, XML converter with templates
Universal JSON, BSON, YAML, CSV, XML converter with templates

Universal JSON, BSON, YAML, CSV, XML translator to ANY format using templates Key features Various input formats (json, bson, yaml, csv, xml) Flexible

Dec 11, 2022
Simple, lightweight and faster response (JSON, JSONP, XML, YAML, HTML, File) rendering package for Go

Package renderer Simple, lightweight and faster response (JSON, JSONP, XML, YAML, HTML, File) rendering package for Go Installation Install the packag

Dec 13, 2022
yaml-patch is a version of Evan Phoenix's json-patch, which is an implementation of JSON Patch, directly transposed to YAML

yaml-patch yaml-patch is a version of Evan Phoenix's json-patch, which is an implementation of JavaScript Object Notation (JSON) Patch, directly trans

Jan 15, 2022
XPath package for Golang, supports HTML, XML, JSON document query.

XPath XPath is Go package provides selecting nodes from XML, HTML or other documents using XPath expression. Implementation htmlquery - an XPath query

Dec 28, 2022