CONFLATE
Library providing routines to merge and validate JSON, YAML, TOML files and/or structs (godoc)
Typical use case: Make your application configuration files multi-format, modular, templated, sparse, location-independent and validated
Description
Conflate is a library and cli-tool, that provides the following features :
- merge data from multiple formats (JSON/YAML/TOML/go structs) and multiple locations (filesystem paths and urls)
- validate the merged data against a JSON schema
- apply any default values defined in a JSON schema to the merged data
- expand environment variables inside the data
- marshal merged data to multiple formats (JSON/YAML/TOML/go structs)
It supports draft-04, draft-06 and draft-07 of JSON Schema. If the key $schema is missing, or the draft version is not explicitly set, a hybrid mode is used which merges together functionality of all drafts into one mode. Improvements, ideas and bug fixes are welcomed.
Getting started
Run the following command, which will build and install the latest binary in $GOPATH/bin
go get github.com/miracl/conflate/...
Alternatively, you can install one of the pre-built release binaries from https://github.com/miracl/conflate/releases
Usage of Library
Please refer to the godoc and the example code
Usage of CLI Tool
Help can be obtained in the usual way :
$conflate --help
Usage of conflate:
-data value
The path/url of JSON/YAML/TOML data, or 'stdin' to read from standard input
-defaults
Apply defaults from schema to data
-expand
Expand environment variables in files
-format string
Output format of the data JSON/YAML/TOML
-includes string
Name of includes array. Blank string suppresses expansion of includes arrays (default "includes")
-noincludes
Switches off conflation of includes. Overrides any --includes setting.
-schema string
The path/url of a JSON v4 schema file
-validate
Validate the data against the schema
-version
Display the version number
To conflate the following file ... :
$cat ./testdata/valid_parent.json
{
"includes": [
"valid_child.json",
"valid_sibling.json"
],
"parent_only" : "parent",
"parent_child" : "parent",
"parent_sibling" : "parent",
"all": "parent"
}
...run the following command, which will merge valid_parent.json, valid_child.json, valid_sibling.json :
$conflate -data ./testdata/valid_parent.json -format JSON
{
"all": "parent",
"child_only": "child",
"parent_child": "parent",
"parent_only": "parent",
"parent_sibling": "parent",
"sibling_child": "sibling",
"sibling_only": "sibling"
}
Note how the includes are loaded remotely as relative paths.
Also, note values in a file override values in any included files, and that an included file overrides values in any included file above it in the includes list.
If you instead host a file somewhere else, then just use a URL :
$conflate -data https://raw.githubusercontent.com/miracl/conflate/master/testdata/valid_parent.json -format JSON
{
"all": "parent",
"child_only": "child",
"parent_child": "parent",
"parent_only": "parent",
"parent_sibling": "parent",
"sibling_child": "sibling",
"sibling_only": "sibling"
}
The includes here are also loaded as relative urls and follow exactly the same merging rules.
To output in a different format use the -format option, e.g. TOML :
$conflate -data ./testdata/valid_parent.json -format TOML
all = "parent"
child_only = "child"
parent_child = "parent"
parent_only = "parent"
parent_sibling = "parent"
sibling_child = "sibling"
sibling_only = "sibling"
To additionally use defaults from a JSON schema and validate the conflated data against the schema, use -defaults and -validate respectively :
$cat ./testdata/blank.yaml
$conflate -data ./testdata/blank.yaml -schema ./testdata/test.schema.json -validate -format YAML
Schema validation failed : The document is not valid against the schema : Invalid type. Expected: object, given: null (#)
$conflate -data ./testdata/blank.yaml -schema ./testdata/test.schema.json -defaults -validate -format YAML
all: parent
child_only: child
parent_child: parent
parent_only: parent
parent_sibling: parent
sibling_child: sibling
sibling_only: sibling
Note any defaults are applied before validation is performed, as you would expect.
If you don't want to intrusively embed an "includes" array inside your JSON, you can instead provide multiple data files which are processed from left-to-right :
$conflate -data ./testdata/valid_child.json -data ./testdata/valid_sibling.json -format JSON
{
"all": "sibling",
"child_only": "child",
"parent_child": "child",
"parent_sibling": "sibling",
"sibling_child": "sibling",
"sibling_only": "sibling"
}
Or alternatively, you can create a top-level JSON file containing only the includes array. For fun, lets choose to use YAML for the top-level file, and output TOML :
$cat toplevel.yaml
includes:
- testdata/valid_child.json
- testdata/valid_sibling.json
$conflate -data toplevel.yaml -format TOML
all = "sibling"
child_only = "child"
parent_child = "child"
parent_sibling = "sibling"
sibling_child = "sibling"
sibling_only = "sibling"
If you want to read a file from stdin you can do the following. Here we pipe in some TOML to override a value to demonstrate :
$echo 'all="MY OVERRIDDEN VALUE"' | conflate -data ./testdata/valid_parent.json -data stdin -format JSON
{
"all": "MY OVERRIDDEN VALUE",
"child_only": "child",
"parent_child": "parent",
"parent_only": "parent",
"parent_sibling": "parent",
"sibling_child": "sibling",
"sibling_only": "sibling"
}
Note that in all cases -data sources are processed from left-to-right, with values in right files overriding values in left files, so the following doesn't work :
$echo 'all="MY OVERRIDDEN VALUE"' | conflate -data stdin -data ./testdata/valid_parent.json -format JSON
{
"all": "parent",
"child_only": "child",
"parent_child": "parent",
"parent_only": "parent",
"parent_sibling": "parent",
"sibling_child": "sibling",
"sibling_only": "sibling"
}
You can optionally expand environment variables in the files like this :
$export echo MYVALUE="some value"
$export echo MYJSONMAP='{ "item1" : "value1" }'
$echo '{ "my_value": "$MYVALUE", "my_map": $MYJSONMAP }' | conflate -data stdin -expand -format JSON
{
"my_map": {
"item1": "value1"
},
"my_value": "some value"
}
Acknowledgements
Images derived from originals by Renee French https://golang.org/doc/gopher/
