Basic TCO support would substantially improve gojq's speed and memory efficiency for cetain jq programs.

This note first considers a simple and direct test of recursion limits (see [*Recursion] below), and then a more practical test involving an optimized form of walk and a well-known large JSON file https://github.com/ryanwholey/jeopardy_bot/blob/master/JEOPARDY_QUESTIONS1.json

[*Recursion]

for jq in gojq ; do
    echo $jq ::
    /usr/bin/time -lp $jq --argjson max 100000000 -n '
    def zero_arity:
      if . == $max then "completed \($max)"
      else if (. % 100000 == 0) then . else empty end, ((.+1)| zero_arity)
      end;
    1 | zero_arity'
done

In abbreviated form, the output for jq is:

"completed 100000000"
user        90.48
sys          0.24
   1880064  maximum resident set size

For gojq, the program fails to complete in a reasonable amount of time. In fact, it takes many hours to reach 600,000.

Setting max to 100,000 gives these performance indicators:

user 78.83 sys 0.65 47173632 maximum resident set size

[*walk]

#!/bin/bash

for jq in jq gojq ; do
    echo $jq
  /usr/bin/time -lp $jq '
  def walk(f):
  def w:
    if type == "object"
    then . as $in
    | reduce keys[] as $key
        ( {}; . + { ($key):  ($in[$key] | w) } ) | f
    elif type == "array" then map( w ) | f
    else f
    end;
  w;

  walk(if type == "string" then 0 else 1 end)
  ' jeopardy.json > /dev/null

done

Results:

jq
user         6.44
sys          0.11
 226742272  maximum resident set size

gojq
user         9.99
sys          0.85
1861201920  maximum resident set size

Hi! this is more of a feature request than a PR but I choose to use a PR to show some proof of concept code.

Background is that i'm working on tool based on gojq that has a interactive CLI REPL interface. The REPL used to be implemented in go with quite a lot of messy code to make it "feel" like jq. Some days I realized that much of this could probably be solved if the REPL itself was written in jq. After some thinking i realized that adding support for custom iterator functions would enable implementing eval as custom function.

With read and print as custom functions you can implement a REPL like this:

def repl:
	def _wrap: if (. | type) != "array" then [.] end;
	def _repl:
		try read("> ") as $e |
		(try (.[] | eval($e)) catch . | print),
		_repl;
	_wrap | _repl;

A bit more complicated than def repl: read | eval(.) | print, repl; repl to make it more user friendly.

Example usage showing basic expressions, nested REPL and errors:

$ go run cmd/repl/main.go
> 2+2
4
> 1 | repl
> .+2
3
> ^D
> [1,2,3] | repl
> .+10
11
12
13
> ^D
> 123
123
> undefined
function not defined: undefined/0
> [1,2,3] | repl
> undefined
function not defined: undefined/0
> ^D
> ^D
$

Some implementation notes:

• The repl takes an array as input and will array wrap non-array values. This feels natural but maybe there are better solutions?
• How to handle 1, 2, 3 | repl. The current behavior to give multiple REPLs feels ok i think.
• Im unsure how to handle env.paths in *env.Next(). I guess it's related to assignment and paths? i haven't looked into how this could affect it.
• The code to implement the iterator can probably be much clear and nicer

What do you think? could be useful?

This was caught out by a test failure on mips64:

--- FAIL: TestCliRun (0.86s)
    --- FAIL: TestCliRun/multiply_strings (0.00s)
        cli_test.go:87: standard output:
              (
              	"""
              	... // 8 identical lines
              	"abcabc"
              	null
            - 	null
            + 	""
              	"""
              )
            
FAIL
FAIL	github.com/itchyny/gojq/cli	0.872s

The result of int(math.Inf(1))¹ depends on the architecture, and is not correct in general to do.

On s390x / ppc64le (big-endian), it returns 9223372036854775807, which would have failed as well, would it not be for the limit condition.

A simple way to fix this is to check:

math.IsNaN(cnt - cnt)

Which catches both infinite (positive / negative) as well as NaN.

The main need is to be able to write a pipeline along the lines of:

"abc" | run("md5")

with the result: "0bee89b07a248e27c83fc3d5951213c1"

Ideally, you could use this in conjunction with try/catch:

"abc" | try system("md5") catch .

This fits in nicely with jq's pipes-and-filters, but of course a reasonable alternative would be to follow the lead of Go's exec.Command(), and have the result be a JSON object with various keys holding the results.

This gojq issue is related to the jq issue at https://github.com/stedolan/jq/issues/147

My impression is that progress on this functionality has stagnated at stedolan/jq mainly because the issue became entangled with numerous other potential enhancements (see especially https://github.com/stedolan/jq/pull/1843).

So my suggestion would be to keep gojq's initial support for "shelling out" quite simple.

Thank you again.

Hello! Would it be possible to add support for own "internal" functions written in go when using gojq as a library? I had a quick look at the code and it seemed not that far away but maybe i'm missing something? If you think it's a good i would even be happy to try implement it if you give some pointers.

Avoids exponential string append

Before: BenchmarkJoinShort BenchmarkJoinShort-4 26947 44399 ns/op BenchmarkJoinLong BenchmarkJoinLong-4 145 7295904 ns/op

After: BenchmarkJoinShort BenchmarkJoinShort-4 81789 14456 ns/op BenchmarkJoinLong BenchmarkJoinLong-4 22002 53791 ns/op

Currently, gojq's builtin.jq follows jq's inefficient defs of index/1 and rindex/1, though jq's builtin.jq has a "TODO:" note about optimization.

The following is a "jq-only" approach to rectifying the problem logically (i.e., without regard to implementation issues arising from memory management). It is relatively complex because it attempts to replicate the existing functionality exactly, and to replicate or improve jq's error messages.

For example, consider the bizarre error message produced by:

jq -n '{a:10} | index("a")'
jq: error (at <unknown>): Cannot index number with number

The new error message would be:

jq: error (at <unknown>): cannot index object with string "a"

index/1

# Both input and $x are assumed to be of the same type, either strings or arrays
def _index_in($x):
  . as $in
  | ($x | length) as $xl
  | first( if type == "string"
           then if $x == "" then null # for conformity with jq
                else range(0; 1 + length - $xl) | select( $in[. : ] | startswith($x))
                end
           else      range(0; 1 + length - $xl) | select( $in[. : .+$xl ] == $x)
           end )
    // null;

def index($x):
  def s: # for the error message
    if type == "object" or type == "array" then ""
    elif type == "string" then " \"\(.)\""
    else " \(.)"
    end;
   def ix($y):
    . as $in
    | first( range(0; length) | select($in[.] == $y) ) // null;
    
  ($x|type) as $xt
  | type as $t
  | if $xt == $t
    then if ($xt == "array" and ($x|length) == 1) then ix($x[0])
         elif $t == "string" or $t == "array" then _index_in($x)
	 else "cannot index \($t) with \($xt)\($x|s)" | error
	 end
    elif $t == "array" then ix($x)
    else "cannot index \($t) with \($xt)\($x|s)" | error
    end ;

Testing and comparing

Here are some test cases, which should be run after changing def index above to def myindex.

def test($in; $x):
  ([$in,$x] | debug) as $debug
  | try ($in | index($x)) catch . ,
    try ($in | myindex($x)) catch .,
    "" ;

test([0,1,2];2),
test([0,1,2];[2]),
test([0,1,2,3,4];[2,3]),
test([0,1,2];[2,3]),
test("abcdef"; []),
test("abcdef"; ""),
test("abcdef"; "c"),
test("abcdef"; "cd"),
test({a:10}; 10),

# jq gives a bizarre error message:
test({a:10}; "a")

Unfortunately, the jq specifications regarding modules are quite complex at best, so here are the relevant bits regarding the default value of the -L path:

The default search path is the search path given to the -L command-line option, else ["~/.jq", "$ORIGIN/../lib/jq", "$ORIGIN/../lib"].

where:

"For paths starting with "$ORIGIN/", the path of the jq executable is substituted for "$ORIGIN"."

To see the issues, consider this transcript:

$ cd ~/github

$ cat lib/jq/foo.jq
def foo: "This is ~/github/lib/jq/foo.jq speaking";

$ cat lib/gojq/foo.jq
def foo: "This is ~/github/lib/gojq/foo";

$ cat lib/gojq/foo.gojq
def foo: "This is ~/github/lib/gojq/foo.gojq";

jq works as advertised:

$ jqMaster/jq -nM 'include "foo"; foo'
"This is ~/github/lib/jq/foo.jq speaking"

$ jqMaster/jq -nM 'include "foo" {"search": "~/github/lib/jq"}; foo'
"This is ~/github/lib/jq/foo.jq speaking"

But ...

$ gojq/gojq -nM 'include "foo"; foo'
gojq: compile error: module not found: "foo"

$ gojq/gojq -nM 'include "foo" {"search": "~/github/lib/gojq"}; foo'
gojq: compile error: module not found: "foo"

Checking the files are still there:

$ file lib/gojq/foo.gojq
lib/gojq/foo.gojq: ASCII text

$ file lib/gojq/foo.jq
file lib/gojq/foo.jq
lib/gojq/foo.jq: ASCII text

So it looks like the "bug" regarding source is real, but maybe gojq's default search path is intentionally different?

The following function definitions exploit the unbounded-precision of integer arithmetic in gojq but are compatible with jq.

That is, if they were added as built-ins to gojq, a stedolan/jq user could simply copy-and-paste them into their programs.

Also, chances are that if filters with the same names and arities as these were added to stedolan/jq in the future, they would have essentially the same semantics, so it is unlikely there would ever be much of a conflict between jq and gojq.

If a user's jq program defines conflicting filters, and if that user used gojq, then the following defs would simply be overridden, so there would be no issue in that case either. Even if such a user wanted to use their private defs and the same-name-same-arity gojq-defined def in the same program, that would be possible, with a tiny amount of effort.

# If $j is 0, then an error condition is raised;
# otherwise, assuming infinite-precision integer arithmetic,
# if the input and $j are integers, then the result will be an integer.
def idivide($j):
  . as $i
  | ($i % $j) as $mod
  | ($i - $mod) / $j ;

# input should be a non-negative integer for accuracy
# but may be any non-negative finite number
def isqrt:
  def irt:
  . as $x
    | 1 | until(. > $x; . * 4) as $q
    | {$q, $x, r: 0}
    | until( .q <= 1;
        .q |= idivide(4)
        | .t = .x - .r - .q
        | .r |= idivide(2)
        | if .t >= 0
          then .x = .t
          | .r += .q
          else .
          end)
    | .r ;
  if type == "number" and (isinfinite|not) and (isnan|not) and . >= 0
  then irt
  else "isqrt requires a non-negative integer for accuracy" | error
  end ;

# It is assumed that $n >= 0 is an integer
# . ^ $n
def power($n):
  . as $a
  | {p:1, $a, $n}
  | until (.n == 0;
       if (.n % 2) == 1 then .p = (.p*.a) else . end
       | .n |= idivide(2)
       | .a |= ((.*.)) )
  | .p;

Accidentally found this difference. Seems like jq floors start and ceils stop index. gojq floors both.

# version 728837713fb78c6ab6d83180e0c7659693f84d25
$ gojq -n '[0,1,2,3][1.9:2.9]'
[
  1
]

# version jq-1.6-159-gcff5336
$ jq -n '[0,1,2,3][1.9:2.9]'
[
  1,
  2
]

jq allows us to use a keyword as a (non-variable) function argument, although I'm not sure if there's a way to use that. gojq on the other hand throws an error.

$ jq -n 'def f(true): true; . | f(.)'
true
$ gojq -n 'def f(true): true; . | f(.)'
gojq: invalid query: def f(true): true; . | f(.)
    def f(true): true; . | f(.)
          ^  unexpected token "true"
[2]    519266 exit 3     gojq -n 'def f(true): true; . | f(.)'
$ gojq --version
gojq 0.12.6 (rev: HEAD/go1.17.6)