utter(github.com/kortschak/utter) is powerful to dump go data structure as go source code.

To dump ast and make the source code usable in other package, we need make "loc" and "freeVariables" in NodeBase struct as public. Otherwise, the dumped source code can only be used in "ast" package.

This pr is just send out for further comment and feedback and demonstrates that pre-parse stdlib is practicable. If this way looks good, I will close this and send several elaborate PRs:

• make "loc" and "freeVariables" in NodeBase struct as public
• add a "Dump()" function in "ast" package, which leveraging utter(github.com/kortschak/utter)
• add a dump command line tool which can read a jsonnet file and dump it's ast. The dump tool will call the "Dump()" function in "ast" package
• using the dump tool to generate go source code for stdlib, and embed it into vm.

Implements: https://github.com/google/go-jsonnet/issues/496

• I reused the fix_plus_sign.go file instead of removing it and creating a new one. Let me know if you would like rename it.
• I don't add new Fodder nor interact with existing Fodder in any way. I think that is correct but that's for someone more knowledgeable to say.
• I'm not sure if the comments are correct so please check them.

This streamlines and simplifies local development by adding a Makefile and moving a majority of commands listed in the README to a Make target.

Additionally,

Add support for asdf-managed versions of bazel

https://github.com/asdf-vm/asdf https://github.com/rajatvig/asdf-bazel

Adds std.parseYaml to address the YAML aspect of: https://github.com/google/jsonnet/issues/460

CPP jsonnet implemented here: https://github.com/google/jsonnet/pull/888

In order to more easily facilitate use of the Go port of the jsonnet tool—instead of the C++ port—from consuming projects like rules_jsonnet, allow this project to be built with Bazel.

There are a few concessions introduced here:

• Generate the standard library AST into a dedicated package (astgen)
  Doing so breaks a circular dependency otherwise caused by generating this source file into the ast package.
• Export all the fields in the struct types exported from the ast package
  Writing the dumped struct initializers into the astgen package requires being able to mention the struct fields defined in the ast package from a separate package (astgen).
• Install the standard library AST into the ast package explicitly
  The AST generated in the astgen package is made available to programs that import it, but it doesn't automatically set this AST as the one used by the ast package. Doing so is possible in an init function in the generated in the astgen package, but setting it up automatically there felt too mysterious. Instead, do it explicitly where necessary: in the jsonnet tool and in the test package that uses it.
• Make fewer assumptions about the right file paths needed by the dumpstdlibast program
  When running the program with Bazel, both its input file path and its output file path differ from the seemingly more obvious paths used when invoking it outside of Bazel.
• Make fewer assumptions about the include file paths used in the c-bindings program
  When building with Bazel, the include file paths are different from the paths used when building outside of Bazel. Use the cgo directives to accommodate both environments.

This is an implementation of https://github.com/google/jsonnet/issues/195 for the Go version since the concerns mentioned in that issue are of no concern in a Go application. It's just a very quick fix done for my own needs, feel free to reject in case you don't want any divergence from the C++ implementation.

go-jsonnet's release packages don't contain cpp-jsonnet/stdlib, but does contain a BUILD file referring to this directory.

This breaks building this repository using go_repository module from rules_go (and also, makes gazelle's autogenerated configuration unusable). There's a workaround proposed by gazelle maintainers (https://github.com/bazelbuild/bazel-gazelle/issues/732#issuecomment-600882662), but this requires manual configuration.

So far I did this in a way which is intended to be minimally invasive, even at the cost of worse performance and some weirdness. Most importantly valueCachedObject should probably replace valueObject interface to avoid some indirection (and it shouldn't be at the same "level" as valueSimpleObject/valueExtendedObject (these should no longer even be values on their own).

This is a reasonable proof of concept, which we can try benchmarking on the real world code.

$ cat foo.jsonnet
local f(x, y = 123, z) = x + y + z;
f(1, 2, 3)

$ jsonnet/bazel-bin/cmd/jsonnet foo.jsonnet
6

$ ./sjsonnet.jar foo.jsonnet
6

$ go-jsonnet/jsonnet  foo.jsonnet
foo.jsonnet:1:21-22 Positional argument after a named argument is not allowed

local f(x, y = 123, z) = x + y + z;

This seems to be a new error in the go-jsonnet implementation. The old C++ google/jsonnet or Scala databricks/sjsonnet don't have any problems with no-default-value params like z following with-default-value params like y. Anyway, the message doesn't make much sense because whether an argument as a default or not, it can always be passed either positionally or via its name.

This is on the following version:

$ go-jsonnet/jsonnet
Jsonnet commandline interpreter v0.13.0

Hi!

In an effort to build a static import analysis tool while creating Tanka (https://github.com/grafana/tanka), we have been using some dirty-patching on jsonnet to speed up analysis speed.

Basically we are implementing a custom Importer that traces its way through the Imports to build a list of transient dependencies after all.

• Import Analysis: https://github.com/grafana/tanka/blob/00ae3bade02f3198585b5f7112c53929d3d3a146/pkg/jsonnet/transitive.go#L8
• TraceImporter: https://github.com/grafana/tanka/blob/00ae3bade02f3198585b5f7112c53929d3d3a146/pkg/jsonnet/jsonnet.go#L78

A full evaluation of jsonnet is not required here, which means we abort early:

https://github.com/sh0rez/go-jsonnet/blob/2787aa93adcc2f2b04e915a8f5523d589ead2ef3/vm.go#L202-L218

However, this patch is definitely not the right way, however I am not familiar enough with the jsonnet codebase to do a better one.

I have seen the efforts in 21c00f1b9ebc856aabf72e2bdb818e1f5b3c11f7 as well, however it is not obvious to me whether I can make use of this.

@sbarzowski Can you provide some details on this, whether I can use the efforts from 21c00f1b9ebc856aabf72e2bdb818e1f5b3c11f7 for my use-case or how to a better patch into go-jsonnet

👋 Hi there, similar to ksonnet, there is interest in the drone community to use jsonnet to simplify pipeline configuration files. Our community is building increasingly complex yaml files like this that could really benefit from jsonnet.

For reference, this is a simple example of a yaml pipeline configuration that defines two pipeline steps, executed sequentially:

pipeline:
  frontend:
    image: node
    commands:
      - npm install
      - npm test
  backend:
    image: golang
    commands:
      - go build
      - go test

Ideally a jsonnet representation would directly mirror the yaml structure, giving developers the ability to more easily migrate from yaml to jsonnet. For example:

{
  frontend: {
    image: "node",
    commands: [
      "npm install",
      "npm test"
     ]
  },
  backend: {
    image: "golang",
    commands: [
      "go build",
      "go test"
     ]
  }
}

The challenge we face is that jsonnet orders keys alphabetically, which means we cannot rely on the ordering of the pipeline steps. Changing our yaml pipeline structure to use a slice instead of a map is unfortunately not an option for us, since it would break a large number of our existing installations.

I was therefore wondering if the community would be open to (optionally) retaining order, perhaps in a manner similar to go-yaml. The go-yaml package implements custom types that retain order when unmarshaled and marshaled:

• https://godoc.org/gopkg.in/yaml.v2#MapItem
• https://godoc.org/gopkg.in/yaml.v2#MapSlice

I do apologize if it seems forward to ask a go-jsonnet to add complexity to its implementation to support the a single project/community, and I certainly understand if such a change/customization is not desired. If there is interest, however, I would happily volunteer to author such a change. Thanks for the consideration!

We're currently working with the length of the string itself, which is the number of bytes. That's wrong in these functions which operate at the level of characters. When applied to a string with a multibyte character they can return the wrong result. For example, evaluating std.rstripChars('• foo\n', '\n') results in "• f".

What we need to do instead is get the number of runes, and remove one rune at a time.

Fixes std.lstripChars and std.rstripChars when the source string contains multibyte (e.g. Unicode) characters.

I added std.stringEscapeXml in github.com/google/jsonnet - found that work is needed to update it here (https://github.com/google/go-jsonnet#updating-and-modifying-the-standard-library)

See

• https://github.com/google/jsonnet/pull/1037
• https://github.com/google/jsonnet/pull/1038

I just found out that a null value is added to the array when I parse a stream of YAML documents that starts with a comment.

YAML

# Test
---
foo: bar
---
baz: cuux

Jsonnet

std.parseYaml(importstr 'test.yaml')

Result

[
   null,
   {
      "foo": "bar"
   },
   {
      "baz": "cuux"
   }
]

Expected Result

[
   {
      "foo": "bar"
   },
   {
      "baz": "cuux"
   }
]

Current Workaround

std.prune(std.parseYaml(importstr 'test.yaml'))

Let's say I have a seet of imports

service = import "./service.jsonnet"
serviceAddOn = import "./service-add-on.jsonnet"

Well, with jsonnefmt it comes out sorted as :

serviceAddOn = import "./service-add-on.jsonnet"
service = import "./service.jsonnet"

This is because "-" comes before "." in the ASCII alphabet. However, if you would simply drop the '.jsonnet' from each file name before sorting them, it would come out in what most people perceive as the correct order :

service = import "./service.jsonnet"
serviceAddOn = import "./service-add-on.jsonnet"

How can we limit the memory/cpu usage when for evaluating jsonnet file? Is it possible withing this library or do we need to use some other library to limit memory/cpu used by go-jsonnet