Cuetils
CUE utilities and helpers for working with tree based objects in any combination of CUE, Yaml, and JSON.
Using
As a command line binary
The cuetils CLI is useful for bulk operations or when you don't want to write extra CUE or Go.
Download a release from GitHub.
cuetils -h
As a Go library
The Go libraries are optimized and have more capabilities.*
go get github.com/hofstadter-io/cuetils@latest
import "github.com/hofstadter-io/cuetils/structural"
* work in progress, unoptimized use the CUE helper and RecurseN
As a CUE library
The CUE libraries all use the RecurseN helper and make use of the "function" pattern. You can also write custom operators.
Add to your project with hof mod or another method. See: https://cuetorials.com/first-steps/modules-and-packages/#dependency-management
import "github.com/hofstadter-io/cuetils/structural"
Structural Helpers
- Count the nodes in an object
- Depth how deep an object is
- Diff two objects, producing a structured diff
- Patch an object, producing a new object
- Pick a subojbect from another, selecting only the parts you want
- Maks a subobject from another, filtering out parts you don't want
- Replace with a subobject, updating fields found
- Upsert with a subobject, updating and adding fields
- Transform one or more objects into another using CUE
- Validate one or more objects with the power of CUE
The helpers work by checking if two operands unify. We try to make note of the edge cases where appropriate, as it depends on both the operation and the method you are using (CUE, Go, or cuetils).
Count
#Count calculates how many nodes are in an object.
CLI example
a: {
foo: "bar"
a: b: c: "d"
}
cow: "moo"
$ cuetils count tree.cue
9
CUE example
import "github.com/hofstadter-io/cuetils/structural"
tree: {
a: {
foo: "bar"
a: b: c: "d"
}
cow: "moo"
}
depth: (structural.#Count & { #in: tree }).out
depth: 9
Go example
Depth
#Depth calculates the deepest branch of an object.
CLI example
a: {
foo: "bar"
a: b: c: "d"
}
cow: "moo"
$ cuetils depth tree.cue
5
CUE example
import "github.com/hofstadter-io/cuetils/structural"
tree: {
a: {
foo: "bar"
a: b: c: "d"
}
cow: "moo"
}
depth: (structural.#Depth & { #in: tree }).out
depth: 5
Go example
import "github.com/hofstadter-io/cuetils/structural"
Diff
#Diff computes a semantic diff object
CLI example
-- a.json --
{
"a": {
"b": "B"
}
}
-- b.yaml --
a:
c: C
b: B
$ cuetils diff a.json b.yaml
{
"+": {
b: "B"
}
a: {
"-": {
b: "B"
}
"+": {
c: "C"
}
}
}
CUE example
import "github.com/hofstadter-io/cuetils/structural"
x: {
a: "a"
b: "b"
d: "d"
e: {
a: "a"
b: "b"
d: "d"
}
}
y: {
b: "b"
c: "c"
d: "D"
e: {
b: "b"
c: "c"
d: 1
}
}
diff: (structural.#Diff & { #X: x, #Y: y }).diff
diff: {
"-": {
a: "a"
d: "d"
}
e: {
"-": {
a: "a"
d: "d"
}
"+": {
d: 1
c: "c"
}
}
"+": {
d: "D"
c: "c"
}
}
Go example
import "github.com/hofstadter-io/cuetils/structural"
For diff and patch, int & 1 like expressions will not be detected. Lists are not currently supported for diff and patch. It may be workable if the list sizes are known and order consistent. Associative Lists may solve this issue. We don't currently have good syntax for specifying the key to match elements on.
Patch
#Patch applies a diff object
CLI example
-- patch.json --
{
"+": {
b: "B"
}
a: {
"-": {
b: "B"
}
"+": {
c: "C"
}
}
}
-- a.json --
{
"a": {
"b": "B"
}
}
$ cuetils patch patch.json a.json
{
b: "B"
a: {
c: "C"
}
}
CUE example
import "github.com/hofstadter-io/cuetils/structural"
x: {
a: "a"
b: "b"
d: "d"
e: {
a: "a"
b: "b"
d: "d"
}
}
p: {
"-": {
a: "a"
d: "d"
}
e: {
"-": {
a: "a"
d: "d"
}
"+": {
d: 1
c: "c"
}
}
"+": {
d: "D"
c: "c"
}
}
patch: (structural.#Patch & { #X: x, #Y: y }).patch
patch: {
b: "b"
c: "c"
d: "D"
e: {
b: "b"
c: "c"
d: 1
}
}
Go example
import "github.com/hofstadter-io/cuetils/structural"
Pick
#Pick extracts a subobject
CLI example
-- pick.cue --
{
a: {
b: string
}
c: int
d: "D"
}
-- a.json --
{
"a": {
"b": "B"
},
"b": 1,
"c": 2,
"d": "D"
}
$ cuetils pick pick.cue a.json
{
a: {
b: "B"
}
c: 2
d: "D"
}
CUE example
import "github.com/hofstadter-io/cuetils/structural"
x: {
a: "a"
b: "b"
d: "d"
e: {
a: "a"
b: "b1"
d: "cd"
}
}
p: {
b: string
d: int
e: {
a: _
b: =~"^b"
d: =~"^d"
}
}
pick: (structural.#Pick & { #X: x, #P: p }).pick
pick: {
b: "b"
e: {
a: "a"
b: "b1"
}
}
Go example
import "github.com/hofstadter-io/cuetils/structural"
Mask
#Mask removes a subobject
-- mask.cue --
{
a: {
b: string
}
c: int
d: "D"
}
-- a.json --
{
"a": {
"b": "B"
"c": "C"
},
"b": 1,
"c": 2,
"d": "D"
}
$ cuetils mask mask.cue a.json
{
a: {
c: "C"
}
b: 1
}
CUE example
import "github.com/hofstadter-io/cuetils/structural"
x: {
a: "a"
b: "b"
d: "d"
e: {
a: "a"
b: "b1"
d: "cd"
}
}
m: {
b: string
d: int
e: {
a: _
b: =~"^b"
d: =~"^d"
}
}
mask: (structural.#Mask & { #X: x, #M: m }).mask
mask: {
a: "a"
d: "d"
e: {
d: "cd"
}
}
Go example
import "github.com/hofstadter-io/cuetils/structural"
Replace
CLI example
-- replace.cue --
{
a: {
b: "b"
}
d: "d"
e: "E"
}
-- a.json --
{
"a": {
"b": "B"
},
"b": 1,
"c": 2,
"d": "D"
}
$ cuetils replace replace.cue a.json
{
b: 1
c: 2
a: {
b: "b"
}
d: "d"
}
CUE example
import "github.com/hofstadter-io/cuetils/structural"
Go example
import "github.com/hofstadter-io/cuetils/structural"
Upsert
CLI example
-- upsert.cue --
{
a: {
b: "b"
}
d: "d"
e: "E"
}
-- a.json --
{
"a": {
"b": "B"
},
"b": 1,
"d": "D"
}
$ cuetils upsert upsert.cue a.json
{
b: 1
a: {
b: "b"
}
d: "d"
e: "E"
}
CUE example
import "github.com/hofstadter-io/cuetils/structural"
Go example
import "github.com/hofstadter-io/cuetils/structural"
Transform
CLI example
-- t.cue --
#In: _ // required, filled in during processing
{
B: #In.a.b
C: #In.a.c
D: #In.d
}
-- a.json --
{
"a": {
"b": "b"
"c": "c"
}
"d": "d"
}
$ cuetils transform t.cue a.json
{
B: "b"
C: "c"
D: "d"
}
Go example
import "github.com/hofstadter-io/cuetils/structural"
Validate
CLI example
-- schema.cue --
{
a: {
b: int
}
c: int
d: "D"
}
-- a.json --
{
"a": {
"b": "B"
},
"b": 1,
"c": 2,
"d": "D"
}
$ cuetils validate schema.cue a.json
a.json
----------------------
a.b: conflicting values "B" and int (mismatched types string and int):
./schema.cue:1:1
./schema.cue:3:6
a.json:3:8
Errors in 1 file(s)
Go example
import "github.com/hofstadter-io/cuetils/structural"
RecurseN
A function factory for bounded recursion, defaulting to 20. This is a pattern to get around CUE's cycle detection by creating a struct with fields named for each iteration. See https://cuetorials.com/deep-dives/recursion/ for more details.
#RecurseN: {
#maxiter: uint | *20
#funcFactory: {
#next: _
#func: _
}
for k, v in list.Range(0, #maxiter, 1) {
#funcs: "\(k)": (#funcFactory & {#next: #funcs["\(k+1)"]}).#func
}
#funcs: "\(#maxiter)": null
#funcs["0"]
}
The core of the bounded recursion is this structural comprehension (unrolled for loop). The "recursive call" is made with the following pattern.
(#funcFactory & {#next: #funcs["\(k+1)"]}).#func
You can override the iterations with { #maxdepth: 100 } at the point of usage or by creating new helpers from the existing ones. You may need to adjust this
- up for deep objects
- down if runtime is an issue
import "github.com/hofstadter-io/cuetils/structural"
#LeaguesDeep: structural.#Depth & { #maxdepth: 10000 }
Custom Helpers
You can make new helpers by building on the #RecurseN pattern. You need two definitions, a factory and the user facing, recursed version.
package structural
import (
"list"
"github.com/hofstadter-io/cuetils/recurse"
)
// A function factory
#depthF: {
// always required
#next: _
// the actual computation, must be named #func
#func: {
// you can have any args
#in: _
// or internal helpers
#multi: {...} | [...]
// the result, can be named anything
depth: {
// detect leafs
if (#in & #multi) == _|_ { 1 }
// detect struct
if (#in & {...}) != _|_ {
list.Max([for k,v in #in {(#next & {#in: v}).depth}]) + 1
}
// detect list
if (#in & [...]) != _|_ {
list.Max([for k,v in #in {(#next & {#in: v}).depth}])
}
}
}
}
// The user facing, recursed version
#Depth: recurse.#RecurseN & {#funcFactory: #depthF}
The core of the recursive calling is:
(#next & {#in: v}).depth
