This an implementation of Jsonnet in pure Go.

go-jsonnet

GoDoc Widget Travis Widget Coverage Status Widget

This an implementation of Jsonnet in pure Go. It is a feature complete, production-ready implementation. It is compatible with the original Jsonnet C++ implementation. Bindings to C and Python are available (but not battle-tested yet).

This code is known to work on Go 1.12 and above. We recommend always using the newest stable release of Go.

Installation instructions

go get github.com/google/go-jsonnet/cmd/jsonnet

It's also available on Homebrew:

brew install go-jsonnet

jsonnetfmt and jsonnet-lint are also available as pre-commit hooks. Example .pre-commit-config.yaml:

- repo: https://github.com/google/go-jsonnet
  rev: # ref you want to point at, e.g. v0.17.0
  hooks:
    - id: jsonnet-format
    - id: jsonnet-lint

It can also be embedded in your own Go programs as a library:

package main

import (
	"fmt"
	"log"

	"github.com/google/go-jsonnet"
)

func main() {
	vm := jsonnet.MakeVM()

	snippet := `{
		person1: {
		    name: "Alice",
		    welcome: "Hello " + self.name + "!",
		},
		person2: self.person1 { name: "Bob" },
	}`

	jsonStr, err := vm.EvaluateAnonymousSnippet("example1.jsonnet", snippet)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(jsonStr)
	/*
	   {
	     "person1": {
	         "name": "Alice",
	         "welcome": "Hello Alice!"
	     },
	     "person2": {
	         "name": "Bob",
	         "welcome": "Hello Bob!"
	     }
	   }
	*/
}

Build instructions (go 1.12+)

git clone [email protected]:google/go-jsonnet.git
cd go-jsonnet
go build ./cmd/jsonnet
go build ./cmd/jsonnetfmt
go build ./cmd/jsonnet-deps

To build with Bazel instead:

git clone [email protected]:google/go-jsonnet.git
cd go-jsonnet
git submodule init
git submodule update
bazel build //cmd/jsonnet
bazel build //cmd/jsonnetfmt
bazel build //cmd/jsonnet-deps

The resulting jsonnet program will then be available at a platform-specific path, such as bazel-bin/cmd/jsonnet/darwin_amd64_stripped/jsonnet for macOS.

Bazel also accommodates cross-compiling the program. To build the jsonnet program for various popular platforms, run the following commands:

Target platform Build command
Current host bazel build //cmd/jsonnet
Linux bazel build [email protected]_bazel_rules_go//go/toolchain:linux_amd64 //cmd/jsonnet
macOS bazel build [email protected]_bazel_rules_go//go/toolchain:darwin_amd64 //cmd/jsonnet
Windows bazel build [email protected]_bazel_rules_go//go/toolchain:windows_amd64 //cmd/jsonnet

For additional target platform names, see the per-Go release definitions here in the rules_go Bazel package.

Additionally if any files were moved around, see the section Keeping the Bazel files up to date.

Building libjsonnet.wasm

GOOS=js GOARCH=wasm go build -o libjsonnet.wasm ./cmd/wasm 

Or if using bazel:

bazel build //cmd/wasm:libjsonnet.wasm

Running tests

./tests.sh  # Also runs `go test ./...`

Running Benchmarks

Method 1

go get golang.org/x/tools/cmd/benchcmp
  1. Make sure you build a jsonnet binary prior to making changes.
go build -o jsonnet-old ./cmd/jsonnet
  1. Make changes (iterate as needed), and rebuild new binary
go build ./cmd/jsonnet
  1. Run benchmark:
# e.g. ./benchmark.sh Builtin
./benchmark.sh <TestNameFilter>

Method 2

  1. get benchcmp
go get golang.org/x/tools/cmd/benchcmp
  1. Make sure you build a jsonnet binary prior to making changes.
make build-old
  1. iterate with (which will also automatically rebuild the new binary ./jsonnet)

replace the FILTER with the name of the test you are working on

FILTER=Builtin_manifestJsonEx make benchmark

Implementation Notes

We are generating some helper classes on types by using http://clipperhouse.github.io/gen/. Do the following to regenerate these if necessary:

go get github.com/clipperhouse/gen
go get github.com/clipperhouse/set
export PATH=$PATH:$GOPATH/bin  # If you haven't already
go generate

Update cpp-jsonnet sub-repo

This repo depends on the original Jsonnet repo. Shared parts include the standard library, headers files for C API and some tests.

You can update the submodule and regenerate dependent files with one command:

./update_cpp_jsonnet.sh

Note: It needs to be run from repo root.

Updating and modifying the standard library

Standard library source code is kept in cpp-jsonnet submodule, because it is shared with Jsonnet C++ implementation.

For performance reasons we perform preprocessing on the standard library, so for the changes to be visible, regeneration is necessary:

go run cmd/dumpstdlibast/dumpstdlibast.go cpp-jsonnet/stdlib/std.jsonnet > astgen/stdast.go

**The

The above command creates the astgen/stdast.go file which puts the desugared standard library into the right data structures, which lets us avoid the parsing overhead during execution. Note that this step is not necessary to perform manually when building with Bazel; the Bazel target regenerates the astgen/stdast.go (writing it into Bazel's build sandbox directory tree) file when necessary.

Keeping the Bazel files up to date

Note that we maintain the Go-related Bazel targets with the Gazelle tool. The Go module (go.mod in the root directory) remains the primary source of truth. Gazelle analyzes both that file and the rest of the Go files in the repository to create and adjust appropriate Bazel targets for building Go packages and executable programs.

After changing any dependencies within the files covered by this Go module, it is helpful to run go mod tidy to ensure that the module declarations match the state of the Go source code. In order to synchronize the Bazel rules with material changes to the Go module, run the following command to invoke Gazelle's update-repos command:

bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=bazel/deps.bzl%jsonnet_go_dependencies

Similarly, after adding or removing Go source files, it may be necessary to synchronize the Bazel rules by running the following command:

bazel run //:gazelle
Owner
Google
Google ❤️ Open Source
Google
Comments
  • Dump ast to source code

    Dump ast to source code

    utter(github.com/kortschak/utter) is powerful to dump go data structure as go source code.

    To dump ast and make the source code usable in other package, we need make "loc" and "freeVariables" in NodeBase struct as public. Otherwise, the dumped source code can only be used in "ast" package.

    This pr is just send out for further comment and feedback and demonstrates that pre-parse stdlib is practicable. If this way looks good, I will close this and send several elaborate PRs:

    • make "loc" and "freeVariables" in NodeBase struct as public
    • add a "Dump()" function in "ast" package, which leveraging utter(github.com/kortschak/utter)
    • add a dump command line tool which can read a jsonnet file and dump it's ast. The dump tool will call the "Dump()" function in "ast" package
    • using the dump tool to generate go source code for stdlib, and embed it into vm.
  • Make jsonnetfmt add plus objects instead of removing them

    Make jsonnetfmt add plus objects instead of removing them

    Implements: https://github.com/google/go-jsonnet/issues/496

    • I reused the fix_plus_sign.go file instead of removing it and creating a new one. Let me know if you would like rename it.
    • I don't add new Fodder nor interact with existing Fodder in any way. I think that is correct but that's for someone more knowledgeable to say.
    • I'm not sure if the comments are correct so please check them.
  • feat: simplify and streamline development

    feat: simplify and streamline development

    This streamlines and simplifies local development by adding a Makefile and moving a majority of commands listed in the README to a Make target.

    Additionally,

    Add support for asdf-managed versions of bazel

    https://github.com/asdf-vm/asdf https://github.com/rajatvig/asdf-bazel

  • Adds std.parseYaml

    Adds std.parseYaml

    Adds std.parseYaml to address the YAML aspect of: https://github.com/google/jsonnet/issues/460

    CPP jsonnet implemented here: https://github.com/google/jsonnet/pull/888

  • Allow building and testing the

    Allow building and testing the "go-jsonnet" project with Bazel

    In order to more easily facilitate use of the Go port of the jsonnet tool—instead of the C++ port—from consuming projects like rules_jsonnet, allow this project to be built with Bazel.

    There are a few concessions introduced here:

    • Generate the standard library AST into a dedicated package (astgen)
      Doing so breaks a circular dependency otherwise caused by generating this source file into the ast package.
    • Export all the fields in the struct types exported from the ast package
      Writing the dumped struct initializers into the astgen package requires being able to mention the struct fields defined in the ast package from a separate package (astgen).
    • Install the standard library AST into the ast package explicitly
      The AST generated in the astgen package is made available to programs that import it, but it doesn't automatically set this AST as the one used by the ast package. Doing so is possible in an init function in the generated in the astgen package, but setting it up automatically there felt too mysterious. Instead, do it explicitly where necessary: in the jsonnet tool and in the test package that uses it.
    • Make fewer assumptions about the right file paths needed by the dumpstdlibast program
      When running the program with Bazel, both its input file path and its output file path differ from the seemingly more obvious paths used when invoking it outside of Bazel.
    • Make fewer assumptions about the include file paths used in the c-bindings program
      When building with Bazel, the include file paths are different from the paths used when building outside of Bazel. Use the cgo directives to accommodate both environments.
  • Automatically create output folders

    Automatically create output folders

    This is an implementation of https://github.com/google/jsonnet/issues/195 for the Go version since the concerns mentioned in that issue are of no concern in a Go application. It's just a very quick fix done for my own needs, feel free to reject in case you don't want any divergence from the C++ implementation.

  • Submodule problems with Gazelle when  simply using go_repository

    Submodule problems with Gazelle when simply using go_repository

    go-jsonnet's release packages don't contain cpp-jsonnet/stdlib, but does contain a BUILD file referring to this directory.

    This breaks building this repository using go_repository module from rules_go (and also, makes gazelle's autogenerated configuration unusable). There's a workaround proposed by gazelle maintainers (https://github.com/bazelbuild/bazel-gazelle/issues/732#issuecomment-600882662), but this requires manual configuration.

  • Object field caching

    Object field caching

    So far I did this in a way which is intended to be minimally invasive, even at the cost of worse performance and some weirdness. Most importantly valueCachedObject should probably replace valueObject interface to avoid some indirection (and it shouldn't be at the same "level" as valueSimpleObject/valueExtendedObject (these should no longer even be values on their own).

    This is a reasonable proof of concept, which we can try benchmarking on the real world code.

  • Positional argument after a named argument is not allowed

    Positional argument after a named argument is not allowed

    $ cat foo.jsonnet
    local f(x, y = 123, z) = x + y + z;
    f(1, 2, 3)
    
    $ jsonnet/bazel-bin/cmd/jsonnet foo.jsonnet
    6
    
    $ ./sjsonnet.jar foo.jsonnet
    6
    
    $ go-jsonnet/jsonnet  foo.jsonnet
    foo.jsonnet:1:21-22 Positional argument after a named argument is not allowed
    
    local f(x, y = 123, z) = x + y + z;
    

    This seems to be a new error in the go-jsonnet implementation. The old C++ google/jsonnet or Scala databricks/sjsonnet don't have any problems with no-default-value params like z following with-default-value params like y. Anyway, the message doesn't make much sense because whether an argument as a default or not, it can always be passed either positionally or via its name.

    This is on the following version:

    $ go-jsonnet/jsonnet
    Jsonnet commandline interpreter v0.13.0
    
  • Static import analysis

    Static import analysis

    Hi!

    In an effort to build a static import analysis tool while creating Tanka (https://github.com/grafana/tanka), we have been using some dirty-patching on jsonnet to speed up analysis speed.

    Basically we are implementing a custom Importer that traces its way through the Imports to build a list of transient dependencies after all.

    • Import Analysis: https://github.com/grafana/tanka/blob/00ae3bade02f3198585b5f7112c53929d3d3a146/pkg/jsonnet/transitive.go#L8
    • TraceImporter: https://github.com/grafana/tanka/blob/00ae3bade02f3198585b5f7112c53929d3d3a146/pkg/jsonnet/jsonnet.go#L78

    A full evaluation of jsonnet is not required here, which means we abort early:

    https://github.com/sh0rez/go-jsonnet/blob/2787aa93adcc2f2b04e915a8f5523d589ead2ef3/vm.go#L202-L218

    However, this patch is definitely not the right way, however I am not familiar enough with the jsonnet codebase to do a better one.

    I have seen the efforts in 21c00f1b9ebc856aabf72e2bdb818e1f5b3c11f7 as well, however it is not obvious to me whether I can make use of this.

    @sbarzowski Can you provide some details on this, whether I can use the efforts from 21c00f1b9ebc856aabf72e2bdb818e1f5b3c11f7 for my use-case or how to a better patch into go-jsonnet

  • Preserve object key ordering when generating output

    Preserve object key ordering when generating output

    👋 Hi there, similar to ksonnet, there is interest in the drone community to use jsonnet to simplify pipeline configuration files. Our community is building increasingly complex yaml files like this that could really benefit from jsonnet.

    For reference, this is a simple example of a yaml pipeline configuration that defines two pipeline steps, executed sequentially:

    pipeline:
      frontend:
        image: node
        commands:
          - npm install
          - npm test
      backend:
        image: golang
        commands:
          - go build
          - go test
    

    Ideally a jsonnet representation would directly mirror the yaml structure, giving developers the ability to more easily migrate from yaml to jsonnet. For example:

    {
      frontend: {
        image: "node",
        commands: [
          "npm install",
          "npm test"
         ]
      },
      backend: {
        image: "golang",
        commands: [
          "go build",
          "go test"
         ]
      }
    }
    

    The challenge we face is that jsonnet orders keys alphabetically, which means we cannot rely on the ordering of the pipeline steps. Changing our yaml pipeline structure to use a slice instead of a map is unfortunately not an option for us, since it would break a large number of our existing installations.

    I was therefore wondering if the community would be open to (optionally) retaining order, perhaps in a manner similar to go-yaml. The go-yaml package implements custom types that retain order when unmarshaled and marshaled:

    • https://godoc.org/gopkg.in/yaml.v2#MapItem
    • https://godoc.org/gopkg.in/yaml.v2#MapSlice

    I do apologize if it seems forward to ask a go-jsonnet to add complexity to its implementation to support the a single project/community, and I certainly understand if such a change/customization is not desired. If there is interest, however, I would happily volunteer to author such a change. Thanks for the consideration!

  • `jsonnet_go_dependencies()` fails if go toolchain has already been registered

    `jsonnet_go_dependencies()` fails if go toolchain has already been registered

    In this commit https://github.com/google/go-jsonnet/commit/1b7cbff4cd267022109868c5fb1953f99e2f8355 jsonnet_go_dependencies() was modified to pass a version parameter to go_register_toolchains(). If a go toolchain has already been registered it results in this error:

    Error in fail: go_register_toolchains: version set after go sdk rule declared (go_sdk)
    

    And according to the docs, setting a version is disallowed if a toolchain has already been registered: https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst#go-register-toolchains

  • Command `jsonnet-deps` is not part of released packages

    Command `jsonnet-deps` is not part of released packages

    jsonnet-deps command is not in the latest released tar archives. (https://github.com/google/go-jsonnet/releases/tag/v0.18.0)

    (Specifically, I have downloaded this archive go-jsonnet_0.18.0_Darwin_x86_64.tar.gz.)

    Is this intentional? Is the recommended way to get jsonnet-deps building from source?

  • Asserted variable considered unused

    Asserted variable considered unused

    Hi, I just bumped into an issue when using

    {   
        local input = [1,2,3],
        local knownItems = [1,2],
        local unknownItems = std.filter(function(i) !(i in knownItems), input),
        assert unknownItems == [] : "unexpected items: %s" % std.join(",",unknownItems)
    }
    

    Linter returns

    Unused variable: unknownItems
    

    Usage of the local variable here is obvious to avoid duplication of the filtering, but it is used just for assertion and validation of input data.

    Would it be possible to also reflect usage in assertion?

  • Add context.Context-aware variants of vm.Evaluate* methods, honor the context during evaluation and manifestation.

    Add context.Context-aware variants of vm.Evaluate* methods, honor the context during evaluation and manifestation.

    The idea here is to be able to put some upper limit on the time it takes to process a jsonnet file (or any other piece of code). Previously, the best bet was to fork+exec with a timeout.

    Please review this commit-by-commit:

    • the first one just mechanically moves arguments around (this was a TODO, I hope I got the idea right)
    • the second adds the methods and the context checks

    The added test verifies that the manifestation respects the context, I don't have an idea about how to test the evaluation case. I'll be grateful for any hints :)

Related tags
Pure Go implementation of jq

gojq Pure Go implementation of jq This is an implementation of jq command written in Go language. You can also embed gojq as a library to your Go prod

Jul 31, 2022
A high-performance, zero allocation, dynamic JSON Threat Protection in pure Go
A high-performance, zero allocation, dynamic JSON Threat  Protection in  pure Go

Package gojtp provides a fast way to validate the dynamic JSON and protect against vulnerable JSON content-level attacks (JSON Threat Protection) based on configured properties.

Jul 27, 2022
jsonpointer - an RFC 6901 implementation for Go

jsonpointer - an RFC 6901 implementation for Go Package jsonpointer provides the ability to resolve, assign, and delete values of any type, including

Jun 13, 2022
COBS implementation in Go (Decoder) and C (Encoder & Decoder) with tests.

COBS Table of Contents About The project COBS Specification Getting Started 3.1. Prerequisites 3.2. Installation 3.3. Roadmap Contributing License Con

May 22, 2022
A fluxcd controller for managing manifests declared in jsonnet

jsonnet-controller A fluxcd controller for managing manifests declared in jsonnet. Kubecfg (and its internal libraries) as well as Tanka-style directo

Jun 21, 2022
A Language Server Protocol (LSP) server for Jsonnet

Jsonnet Language Server Warning: This project is in active development and is likely very buggy. A Language Server Protocol (LSP) server for Jsonnet.

Apr 29, 2022
Generate Jsonnet definition for JSON representation of protobuf object

Generate Jsonnet definition for JSON representation of protobuf object

Nov 1, 2021
A Language Server Protocol (LSP) server for Jsonnet

Jsonnet Language Server A Language Server Protocol (LSP) server for Jsonnet. Features Jump to definition self-support.mp4 dollar-support.mp4 Error/War

Aug 3, 2022
Pure Go termbox implementation

IMPORTANT This library is somewhat not maintained anymore. But I'm glad that it did what I wanted the most. It moved people away from "ncurses" mindse

Aug 1, 2022
The pure golang implementation of nanomsg (version 1, frozen)
The pure golang implementation of nanomsg (version 1, frozen)

mangos NOTE: This is the legacy version of mangos (v1). Users are encouraged to use mangos v2 instead if possible. No further development is taking pl

Jun 22, 2022
Pure Go implementation of D. J. Bernstein's cdb constant database library.

Pure Go implementation of D. J. Bernstein's cdb constant database library.

Jun 12, 2022
A QUIC implementation in pure go
A QUIC implementation in pure go

A QUIC implementation in pure Go quic-go is an implementation of the QUIC protocol in Go. It implements the IETF QUIC draft-29 and draft-32. Version c

Aug 1, 2022
Pure Go implementation of the WebRTC API
Pure Go implementation of the WebRTC API

Pion WebRTC A pure Go implementation of the WebRTC API New Release Pion WebRTC v3.0.0 has been released! See the release notes to learn about new feat

Aug 4, 2022
A Windows named pipe implementation written in pure Go.

npipe Package npipe provides a pure Go wrapper around Windows named pipes. Windows named pipe documentation: http://msdn.microsoft.com/en-us/library/w

Jul 27, 2022
mangos is a pure Golang implementation of nanomsg's "Scalablilty Protocols"
mangos is a pure Golang implementation of nanomsg's

mangos Mangos™ is an implementation in pure Go of the SP (“Scalability Protocols”) messaging system. These are colloquially known as a “nanomsg”. ❗ Th

Aug 6, 2022
Pure Go implementation of the NaCL set of API's

go-nacl This is a pure Go implementation of the API's available in NaCL: https://nacl.cr.yp.to. Compared with the implementation in golang.org/x/crypt

Aug 5, 2022
Package git provides an incomplete pure Go implementation of Git core methods.

git Package git provides an incomplete pure Go implementation of Git core methods. Example Code: store := git.TempStore() defer os.RemoveAll(string(st

Mar 1, 2022
Pure Go implementation of the WebRTC API
Pure Go implementation of the WebRTC API

Pure Go implementation of the WebRTC API

Aug 8, 2022
Pure Go implementation of the WebRTC API
Pure Go implementation of the WebRTC API

Pion WebRTC A pure Go implementation of the WebRTC API New Release Pion WebRTC v3.0.0 has been released! See the release notes to learn about new feat

Aug 9, 2022
A Blurhash implementation in pure Go (Decode/Encode)
A Blurhash implementation in pure Go (Decode/Encode)

go-blurhash go-blurhash is a pure Go implementation of the BlurHash algorithm, which is used by Mastodon an other Fediverse software to implement a sw

Jul 3, 2022