Light weight, extensible configuration management library for Go. Built in support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.

koanf

koanf (pronounced conf; a play on the Japanese Koan) is a library for reading configuration from different sources in different formats in Go applications. It is a cleaner, lighter alternative to spf13/viper with better abstractions and extensibility and fewer dependencies.

koanf comes with built in support for reading configuration from files, command line flags, and environment variables, and can parse JSON, YAML, TOML, and Hashicorp HCL.

Build Status GoDoc

Installation

go get -u github.com/knadh/koanf

Contents

Concepts

  • koanf.Provider is a generic interface that provides configuration, for example, from files, environment variables, HTTP sources, or anywhere. The configuration can either be raw bytes that a parser can parse, or it can be a nested map[string]interface{} that can be directly loaded.
  • koanf.Parser is a generic interface that takes raw bytes, parses, and returns a nested map[string]interface{} representation. For example, JSON and YAML parsers.
  • Once loaded into koanf, configuration are values queried by a delimited key path syntax. eg: app.server.port. Any delimiter can be chosen.
  • Configuration from multiple sources can be loaded and merged into a koanf instance, for example, load from a file first and override certain values with flags from the command line.

With these two interface implementations, koanf can obtain a configuration from multiple sources and parse any format and make it available to an application.

Reading config from files

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/parsers/yaml"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load YAML config and merge into the previously loaded config (because we can).
	k.Load(file.Provider("mock/mock.yaml"), yaml.Parser())

	fmt.Println("parent's name is = ", k.String("parent1.name"))
	fmt.Println("parent's ID is = ", k.Int("parent1.id"))
}

Watching files for changes

The koanf.Provider interface has a Watch(cb) method that asks a provider to watch for changes and trigger the given callback that can live reload the configuration.

Currently, file.Provider supports this.

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/parsers/yaml"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	f := file.Provider("mock/mock.json")
	if err := k.Load(f, json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load YAML config and merge into the previously loaded config (because we can).
	k.Load(file.Provider("mock/mock.yaml"), yaml.Parser())

	fmt.Println("parent's name is = ", k.String("parent1.name"))
	fmt.Println("parent's ID is = ", k.Int("parent1.id"))

	// Watch the file and get a callback on change. The callback can do whatever,
	// like re-load the configuration.
	// File provider always returns a nil `event`.
	f.Watch(func(event interface{}, err error) {
		if err != nil {
			log.Printf("watch error: %v", err)
			return
		}

		log.Println("config changed. Reloading ...")
		k.Load(f, json.Parser())
		k.Print()
	})

	// Block forever (and manually make a change to mock/mock.json) to
	// reload the config.
	log.Println("waiting forever. Try making a change to mock/mock.json to live reload")
	<-make(chan bool)
}

Reading from command line

The following example shows the use of posflag.Provider, a wrapper over spf13/pflag library, an advanced commandline lib. For Go's built in flag package, use basicflag.Provider.

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/toml"
	"github.com/knadh/koanf/providers/file"
	"github.com/knadh/koanf/providers/posflag"
	flag "github.com/spf13/pflag"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Use the POSIX compliant pflag lib instead of Go's flag lib.
	f := flag.NewFlagSet("config", flag.ContinueOnError)
	f.Usage = func() {
		fmt.Println(f.FlagUsages())
		os.Exit(0)
	}
	// Path to one or more config files to load into koanf along with some config params.
	f.StringSlice("conf", []string{"mock/mock.toml"}, "path to one or more .toml config files")
	f.String("time", "2020-01-01", "a time string")
	f.String("type", "xxx", "type of the app")
	f.Parse(os.Args[1:])

	// Load the config files provided in the commandline.
	cFiles, _ := f.GetStringSlice("conf")
	for _, c := range cFiles {
		if err := k.Load(file.Provider(c), toml.Parser()); err != nil {
			log.Fatalf("error loading file: %v", err)
		}
	}

	// "time" and "type" may have been loaded from the config file, but
	// they can still be overridden with the values from the command line.
	// The bundled posflag.Provider takes a flagset from the spf13/pflag lib.
	// Passing the Koanf instance to posflag helps it deal with default command
	// line flag values that are not present in conf maps from previously loaded
	// providers.
	if err := k.Load(posflag.Provider(f, ".", k), nil); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	fmt.Println("time is = ", k.String("time"))
}

Reading environment variables

package main

import (
	"fmt"
	"log"
	"strings"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/env"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use . as the key path delimiter. This can be / or anything.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load environment variables and merge into the loaded config.
	// "MYVAR" is the prefix to filter the env vars by.
	// "." is the delimiter used to represent the key hierarchy in env vars.
	// The (optional, or can be nil) function can be used to transform
	// the env var names, for instance, to lowercase them.
	//
	// For example, env vars: MYVAR_TYPE and MYVAR_PARENT1_CHILD1_NAME
	// will be merged into the "type" and the nested "parent1.child1.name"
	// keys in the config file here as we lowercase the key, 
	// replace `_` with `.` and strip the MYVAR_ prefix so that 
	// only "parent1.child1.name" remains.
	k.Load(env.Provider("MYVAR_", ".", func(s string) string {
		return strings.Replace(strings.ToLower(
			strings.TrimPrefix(s, "MYVAR_")), "_", ".", -1)
	}), nil)

	fmt.Println("name is = ", k.String("parent1.child1.name"))
}

You can also use the env.ProviderWithValue with a callback that supports mutating both the key and value to return types other than a string.

	// Load environment variables and merge into the loaded config.
	// "MYVAR" is the prefix to filter the env vars by just as the example above but
	// in this case we check the existing type of the value (if it exists)
	// If it's a slice, we convert the environment variable to a string slice
	k.Load(env.ProviderWithValue("MYVAR_", ".", func(s string, v string) (string, interface{}) {
		// Convert the environment variable key to the koanf key format
		key := strings.Replace(strings.ToLower(strings.TrimPrefix(s, "MYVAR_")), "_", ".", -1)
		// Check to exist if we have a configuration option already and see if it's a slice
		switch k.Get(key).(type) {
		case []interface{}, []string:
			// Convert our environment variable to a slice by splitting on space
			return key, strings.Split(v, " ")
		}
		return key, s // Otherwise return the new key with the unaltered value
	}), nil)
}

Reading from an S3 bucket

// Load JSON config from s3.
if err := k.Load(s3.Provider(s3.Config{
	AccessKey: os.Getenv("AWS_S3_ACCESS_KEY"),
	SecretKey: os.Getenv("AWS_S3_SECRET_KEY"),
	Region:    os.Getenv("AWS_S3_REGION"),
	Bucket:    os.Getenv("AWS_S3_BUCKET"),
	ObjectKey: "dir/config.json",
}), json.Parser()); err != nil {
	log.Fatalf("error loading config: %v", err)
}

Reading raw bytes

The bundled rawbytes Provider can be used to read arbitrary bytes from a source, like a database or an HTTP call.

package main

import (
	"fmt"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/rawbytes"
)

// Global koanf instance. Use . as the key path delimiter. This can be / or anything.
var k = koanf.New(".")

func main() {
	b := []byte(`{"type": "rawbytes", "parent1": {"child1": {"type": "rawbytes"}}}`)
	k.Load(rawbytes.Provider(b), json.Parser())
	fmt.Println("type is = ", k.String("parent1.child1.type"))
}

Unmarshalling and marshalling

Parsers can be used to unmarshal and scan the values in a Koanf instance into a struct based on the field tags, and to marshal a Koanf instance back into serialized bytes, for example, back to JSON or YAML, to write back to files.

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use . as the key path delimiter. This can be / or anything.
var (
	k      = koanf.New(".")
	parser = json.Parser()
)

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), parser); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Structure to unmarshal nested conf to.
	type childStruct struct {
		Name       string            `koanf:"name"`
		Type       string            `koanf:"type"`
		Empty      map[string]string `koanf:"empty"`
		GrandChild struct {
			Ids []int `koanf:"ids"`
			On  bool  `koanf:"on"`
		} `koanf:"grandchild1"`
	}

	var out childStruct

	// Quick unmarshal.
	k.Unmarshal("parent1.child1", &out)
	fmt.Println(out)

	// Unmarshal with advanced config.
	out = childStruct{}
	k.UnmarshalWithConf("parent1.child1", &out, koanf.UnmarshalConf{Tag: "koanf"})
	fmt.Println(out)

	// Marshal the instance back to JSON.
	// The paser instance can be anything, eg: json.Paser(), yaml.Parser() etc.
	b, _ := k.Marshal(parser)
	fmt.Println(string(b))
}

Unmarshalling with flat paths

Sometimes it is necessary to unmarshal an assortment of keys from various nested structures into a flat target structure. This is possible with the UnmarshalConf.FlatPaths flag.

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use . as the key path delimiter. This can be / or anything.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	type rootFlat struct {
		Type                        string            `koanf:"type"`
		Empty                       map[string]string `koanf:"empty"`
		Parent1Name                 string            `koanf:"parent1.name"`
		Parent1ID                   int               `koanf:"parent1.id"`
		Parent1Child1Name           string            `koanf:"parent1.child1.name"`
		Parent1Child1Type           string            `koanf:"parent1.child1.type"`
		Parent1Child1Empty          map[string]string `koanf:"parent1.child1.empty"`
		Parent1Child1Grandchild1IDs []int             `koanf:"parent1.child1.grandchild1.ids"`
		Parent1Child1Grandchild1On  bool              `koanf:"parent1.child1.grandchild1.on"`
	}

	// Unmarshal the whole root with FlatPaths: True.
	var o1 rootFlat
	k.UnmarshalWithConf("", &o1, koanf.UnmarshalConf{Tag: "koanf", FlatPaths: true})
	fmt.Println(o1)

	// Unmarshal a child structure of "parent1".
	type subFlat struct {
		Name                 string            `koanf:"name"`
		ID                   int               `koanf:"id"`
		Child1Name           string            `koanf:"child1.name"`
		Child1Type           string            `koanf:"child1.type"`
		Child1Empty          map[string]string `koanf:"child1.empty"`
		Child1Grandchild1IDs []int             `koanf:"child1.grandchild1.ids"`
		Child1Grandchild1On  bool              `koanf:"child1.grandchild1.on"`
	}

	var o2 subFlat
	k.UnmarshalWithConf("parent1", &o2, koanf.UnmarshalConf{Tag: "koanf", FlatPaths: true})
	fmt.Println(o2)
}

Marshalling and writing config

It is possible to marshal and serialize the conf map into TOML, YAML etc.

Setting default values.

koanf does not provide any special functions to set default values but uses the Provider interface to enable it.

From a map

The bundled confmap provider takes a map[string]interface{} that can be loaded into a koanf instance.

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/providers/confmap"
	"github.com/knadh/koanf/providers/file"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/parsers/yaml"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Load default values using the confmap provider.
	// We provide a flat map with the "." delimiter.
	// A nested map can be loaded by setting the delimiter to an empty string "".
	k.Load(confmap.Provider(map[string]interface{}{
		"parent1.name": "Default Name",
		"parent3.name": "New name here",
	}, "."), nil)

	// Load JSON config on top of the default values.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load YAML config and merge into the previously loaded config (because we can).
	k.Load(file.Provider("mock/mock.yaml"), yaml.Parser())

	fmt.Println("parent's name is = ", k.String("parent1.name"))
	fmt.Println("parent's ID is = ", k.Int("parent1.id"))
}

From a struct

The bundled structs provider can be used to read data from a struct to load into a koanf instance.

package main

import (
	"fmt"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/providers/structs"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

type parentStruct struct {
	Name   string      `koanf:"name"`
	ID     int         `koanf:"id"`
	Child1 childStruct `koanf:"child1"`
}
type childStruct struct {
	Name        string            `koanf:"name"`
	Type        string            `koanf:"type"`
	Empty       map[string]string `koanf:"empty"`
	Grandchild1 grandchildStruct  `koanf:"grandchild1"`
}
type grandchildStruct struct {
	Ids []int `koanf:"ids"`
	On  bool  `koanf:"on"`
}
type sampleStruct struct {
	Type    string            `koanf:"type"`
	Empty   map[string]string `koanf:"empty"`
	Parent1 parentStruct      `koanf:"parent1"`
}

func main() {
	// Load default values using the structs provider.
	// We provide a struct along with the struct tag `koanf` to the
	// provider.
	k.Load(structs.Provider(sampleStruct{
		Type:  "json",
		Empty: make(map[string]string),
		Parent1: parentStruct{
			Name: "parent1",
			ID:   1234,
			Child1: childStruct{
				Name:  "child1",
				Type:  "json",
				Empty: make(map[string]string),
				Grandchild1: grandchildStruct{
					Ids: []int{1, 2, 3},
					On:  true,
				},
			},
		},
	}, "koanf"), nil)

	fmt.Printf("name is = `%s`\n", k.String("parent1.child1.name"))
}

Order of merge and key case senstivity

  • Config keys are case-sensitive in koanf. For example, app.server.port and APP.SERVER.port are not the same.
  • koanf does not impose any ordering on loading config from various providers. Every successive Load() or Load() merges new config into existing config. That means it is possible to load environment variables first, then files on top of it, and then command line variables on top of it, or any such order.

Custom Providers and Parsers

A Provider can provide a nested map[string]interface{} config that can be loaded into koanf with koanf.Load() or raw bytes that can be parsed with a Parser (loaded using koanf.Load().

Writing Providers and Parsers are easy. See the bundled implementations in the providers and parses directory.

API

Bundled providers

Package Provider Description
providers/file file.Provider(filepath string) Reads a file and returns the raw bytes to be parsed.
providers/basicflag basicflag.Provider(f *flag.FlagSet, delim string) Takes an stdlib flag.FlagSet
providers/posflag posflag.Provider(f *pflag.FlagSet, delim string) Takes an spft3/pflag.FlagSet (advanced POSIX compatible flags with multiple types) and provides a nested config map based on delim.
providers/env env.Provider(prefix, delim string, f func(s string) string) Takes an optional prefix to filter env variables by, an optional function that takes and returns a string to transform env variables, and returns a nested config map based on delim.
providers/confmap confmap.Provider(mp map[string]interface{}, delim string) Takes a premade map[string]interface{} conf map. If delim is provided, the keys are assumed to be flattened, thus unflattened using delim.
providers/structs structs.Provider(s interface{}, tag string) Takes a struct and struct tag.
providers/s3 s3.Provider(s3.S3Config{}) Takes a s3 config struct.
providers/rawbytes rawbytes.Provider(b []byte) Takes a raw []byte slice to be parsed with a koanf.Parser
providers/vault vault.Provider(vault.Config{}) Hashicorp Vault provider

Bundled parsers

Package Parser Description
parsers/json json.Parser() Parses JSON bytes into a nested map
parsers/yaml yaml.Parser() Parses YAML bytes into a nested map
parsers/toml toml.Parser() Parses TOML bytes into a nested map
parsers/dotenv dotenv.Parser() Parses DotEnv bytes into a flat map
parsers/hcl hcl.Parser(flattenSlices bool) Parses Hashicorp HCL bytes into a nested map. flattenSlices is recommended to be set to true. Read more.

Instance functions

Method Description
Load(p Provider, pa Parser) error Loads config from a Provider. If a koanf.Parser is provided, the config is assumed to be raw bytes that's then parsed with the Parser.
Keys() []string Returns the list of flattened key paths that can be used to access config values
KeyMap() map[string][]string Returns a map of all possible key path combinations possible in the loaded nested conf map
All() map[string]interface{} Returns a flat map of flattened key paths and their corresponding config values
Raw() map[string]interface{} Returns a copy of the raw nested conf map
Print() Prints a human readable copy of the flattened key paths and their values for debugging
Sprint() Returns a human readable copy of the flattened key paths and their values for debugging
Cut(path string) *Koanf Cuts the loaded nested conf map at the given path and returns a new Koanf instance with the children
Copy() *Koanf Returns a copy of the Koanf instance
Merge(*Koanf) Merges the config map of a Koanf instance into the current instance
Delete(path string) Delete the value at the given path, and does nothing if path doesn't exist.
MergeAt(in *Koanf, path string) Merges the config map of a Koanf instance into the current instance, at the given key path.
Unmarshal(path string, o interface{}) error Scans the given nested key path into a given struct (like json.Unmarshal) where fields are denoted by the koanf tag
UnmarshalWithConf(path string, o interface{}, c UnmarshalConf) error Like Unmarshal but with customizable options

Getter functions

Get(path string) interface{} Returns the value for the given key path, and if it doesn’t exist, returns nil
Exists(path string) bool Returns true if the given key path exists in the conf map
Int64(path string) int64
Int64s(path string) []int64
Int64Map(path string) map[string]int64
Int(path string) int
Ints(path string) []int
IntMap(path string) map[string]int
Float64(path string) float64
Float64s(path string) []float64
Float64Map(path string) map[string]float64
Duration(path string) time.Duration Returns the time.Duration value of the given key path if it’s numeric (attempts a parse+convert if string) or a string representation like "3s".
Time(path, layout string) time.Time Parses the string value of the the given key path with the given layout format and returns time.Time. If the key path is numeric, treats it as a UNIX timestamp and returns its time.Time.
String(path string) string
Strings(path string) []string
StringMap(path string) map[string]string
StringsMap(path string) map[string][]string
Byte(path string) []byte
Bool(path string) bool
Bools(path string) []bool
BoolMap(path string) map[string]bool
MapKeys(path string) []string Returns the list of keys in any map
Slices(path string) []Koanf Returns []map[string]interface{}, a slice of confmaps loaded into a slice of new Koanf instances.

Alternative to viper

koanf is a lightweight alternative to the popular spf13/viper. It was written as a result of multiple stumbling blocks encountered with some of viper's fundamental flaws.

  • viper breaks JSON, YAML, TOML, HCL language specs by forcibly lowercasing keys.
  • Significantly bloats build sizes.
  • Tightly couples config parsing with file extensions.
  • Has poor semantics and abstractions. Commandline, env, file etc. and various parses are hardcoded in the core. There are no primitives that can be extended.
  • Pulls a large number of third party dependencies into the core package. For instance, even if you do not use YAML or flags, the dependencies are still pulled as a result of the coupling.
  • Imposes arbitrary ordering conventions (eg: flag -> env -> config etc.)
  • Get() returns references to slices and maps. Mutations made outside change the underlying values inside the conf map.
  • Does non-idiomatic things such as throwing away O(1) on flat maps.
  • There are a large number of open issues.
Owner
Kailash Nadh
CTO @zerodhatech | Volunteer @fossunited
Kailash Nadh
Comments
  • Add ability to escape delimters.

    Add ability to escape delimters.

    Through this PR, you can escape the delim in the key, like this:

    map.Unflatten(map[string]interface{}{"this.is..an..escaped.key": true}, ".", "..") 
    

    will give you

    map[string]interface{}{
      "this": map[string]interface{}{
        "is.an.escaped": map[string]interface{}{
          "key": true,
        },
      },
    }
    
  • Reconsider internal use of `keyMap`

    Reconsider internal use of `keyMap`

    We are using koanf in production for a multi-tenant system and are having serious issues with memory consumption when a number (~5000) of configs is loaded that have a medium amount of config values (~85 flattened key/value pairs) with memory spikes of up to 7 GB.

    Benchmarking and memory profiling shows that a lot of time is spent in maps.Flatten and koanf.populateKeyParts hence my two recent PRs:

    Bildschirmfoto 2021-06-07 um 13 34 44

    I think some of this boils down to KeyMap which contains the flattened keys in string and slice form. I looked through the code and it appears that the feature is not really used, or where it is used, strings.Split could be used instead which would probably reduce the memory count quite a bit because we avoid keeping these in memory:

    koanf.keyMap = map[string][]string{
      "foo": []string{"foo"},
      "foo.bar": []string{"foo", "bar"},
      "foo.bar.baz1": []string{"foo", "bar", "baz1"},
      "foo.bar.baz2": []string{"foo", "bar", "baz2"},
      "foo.bar.baz3": []string{"foo", "bar", "baz3"},
    }
    

    As you can see, the amount of memory explodes the more nested keys are used because the same keyparts are copied over and over. As Koanf strictly uses slice copying to avoid modification (I assume?) this leads to a lot of unnecessary memory use.

    The performance impact of using strings.Split("foo.bar.baz3", delimiter) should be negligible compared to the amount of data kept in memory.

    The only place where we would need to generate this keymap as it exists right now would be the KeyMap() function if we want to keep compatibility.

  • Question about race condition

    Question about race condition

    Hi,

    I am wondering whether we may have a race condition when the Watch method of a provider is called. I imagine that something wrong may happen when the configuration is read and reloaded at the same time. If it's possible, how would you advise me to make it thread safe?

    Thanks a lot

  • posflag: add support for flags that implement the flag.Getter interface from stdlib

    posflag: add support for flags that implement the flag.Getter interface from stdlib

    A proposed implementation of

    • https://github.com/knadh/koanf/issues/102

    Let me know what you think of this approach. If it looks acceptable, I can add tests and docs.

  • Remove keyMap internal from Koanf

    Remove keyMap internal from Koanf

    As outlined in #80, Koanf keeps track of KeyMap internally even though the representation is not really needed. KeyMaps use a significant amount of memory depending on the config layout which can lead to excessive memory consumptions for large or complex configs.

    This patch removes the need to keep keyMap around and therefore significantly improves performance:

    benchmark                old ns/op     new ns/op     delta
    BenchmarkLoadFile-16     146821        135027        -8.03%
    
    benchmark                old allocs     new allocs     delta
    BenchmarkLoadFile-16     679            657            -3.24%
    
    benchmark                old bytes     new bytes     delta
    BenchmarkLoadFile-16     46277         37019         -20.01%
    

    BREAKING CHANGE: This PR deprecates KeyMap function. It still works as before but is slower.

    It also removes the second return argument of maps.Flatten which was the KeyMap previously.

  • Default empty []string unmarshalls as a nil

    Default empty []string unmarshalls as a nil

    When I have a string slice that defaults (via flags) to empty slice ([]string) then it gets unmarshalled as nil.

    This isn't really a bug in koanf but in https://github.com/mitchellh/mapstructure/issues/146 Reporting here since koanf uses mapstructure directly. Is there any way to workaround this in koanf? mitchellh/mapstructure didn't get any commit in a while so looks like the issue might not get fixed, in that case is there an alternative koanf could use?

  • Custom error handler

    Custom error handler

    When validating type of values passed via flags or config file I have the choice of either k.Int() or k.MustInt(), first one will default to 0 if the value is incorrect, second will panic on parse error. What I'm missing is a way of validating passed values and reporting an error to the user without the need to panic, there is no variant of a getter that returns an error from what I see. Is it possible to setup custom error handler? Perhaps one could register error handling function that's called on MustInt() to avoid polluting API with yet another version of a Int() getter?

  • Watch example in README does not remove keys / values

    Watch example in README does not remove keys / values

    The current README example around file watching could be improved by making the limitations of that approach clear. Removing a value from the config file (e.g. parent1.id) will not remove the value from the config but keep the old value around. It would probably be expected for the removed value to be no longer available in the config as it was removed from the file.

    Example

    Run program and have mock.json with:

    {
      "parent1": {
        "name": "foo",
        "id": 34
      }
    }
    
    % go run .
    parent's name is =  bar
    parent's ID is =  34
    

    Remove all keys from mock.json

    {}
    

    Config reloads, but values are still present:

    2021/06/05 17:43:19 config changed. Reloading ...
    parent1.id -> 34
    parent1.name -> bar
    
  • How to provide string slice from environment variables

    How to provide string slice from environment variables

    Any thoughts on how to provide a string slice from an environment variable? Viper looks at the existing value and if the default is a slice it interprets the environment variable as a slice. It almost would have been better to return (string, interface{}) from the callback and allow the callback to manipulate the value as well.

  • Values not being loaded

    Values not being loaded

    I'm working with koanf and it looks like maps and some custom values are not being loaded properly. They appear to be empty even if the toml config file I'm using has values specified.

    Is there a delay or similar on reading values from a config file?

  • unable to parse int64 with leading zeros

    unable to parse int64 with leading zeros

    Describe the bug Looks like koanf uses the standard go yaml.v2 library to parse YAML, but it is behaving differently than expected. Specifically, int64 values with leading zeros are not being parsed/converted correctly and it is throwing an error.

    To Reproduce sample code:

    package main
    
    import (
    	"log"
    
    	"github.com/davecgh/go-spew/spew"
    	"github.com/knadh/koanf"
    	"github.com/knadh/koanf/parsers/yaml"
    	"github.com/knadh/koanf/providers/rawbytes"
    	goyaml "gopkg.in/yaml.v2"
    )
    
    type Example struct {
    	Config map[string]*SubConfig `koanf:"Config" yaml:"Config"`
    }
    
    type SubConfig struct {
    	Accounts map[int64]string `koanf:"Accounts" yaml:"Accounts"`
    }
    
    var data = `
    Config:
      Foo:
        Accounts:
          0012345678912: FancyBear
    `
    
    func main() {
    	// plain yaml decode which works fine
    	example := Example{}
    	if err := goyaml.Unmarshal([]byte(data), &example); err != nil {
    		log.Fatalf("error: %v", err)
    	}
    	log.Printf("yaml decode: %s", spew.Sdump(example))
    
    	// koanf decode whieh is fatal
    	konf := koanf.New(".")
    	s := &Example{}
    	if err := konf.Load(rawbytes.Provider([]byte(data)), yaml.Parser()); err != nil {
    		log.Fatalf("Unable to load data: %s", err.Error())
    	}
    
    	if err := konf.Unmarshal("", s); err != nil {
    		log.Fatalf("Unable to unmarshal data: %s", err.Error())
    	}
    }
    
    $ go run .
    2022/02/23 20:04:38 yaml decode: (main.Example) {
     Config: (map[string]*main.SubConfig) (len=1) {
      (string) (len=3) "Foo": (*main.SubConfig)(0xc000010088)({
       Accounts: (map[int64]string) (len=1) {
        (int64) 12345678912: (string) (len=9) "FancyBear"
       }
      })
     }
    }
    2022/02/23 20:04:38 Unable to unmarshal data: 1 error(s) decoding:
    
    * cannot parse 'Config[Foo].Accounts[1.2345678912e+10]' as int: strconv.ParseInt: parsing "1.2345678912e+10": invalid syntax
    exit status 1
    

    Expected behavior Decode successfully and convert 0012345678912 => 12345678912

    Please provide the following information):

    • OS: macOS
    • Koanf Version 1.4.0

    Additional context

    I get a slightly different error if I simplify the Example struct above into just:

    type Example map[int64]string

    which generates:

    * cannot parse '[0012345678912]' as int: strconv.ParseInt: parsing "0012345678912": invalid syntax

  • unmarshall dict key to struct property

    unmarshall dict key to struct property

    Hey guys, I am new to Go and don't know exactly if this is a yaml or koanf question, so sorry in advance if this is not the right place 🙈 .

    I am trying to unmarshall yaml dict keys onto a property of a struct rather than the key of a map. Given this yaml

    commands:
        php:
            service: php
            bin: /bin/php
        node:
            service: node
            bin: /bin/node
    

    I am able to unmarshall this into a struct like this:

    type Config struct {
    	Commands map[string]struct {
    		Service string
    		Bin     string
    	}
    }
    

    But how am I able to unmarshall it into a struct like this:

    type Config struct {
    	Commands []struct {
    		Name    string    // <-- this should be key from the yaml (i.e. php or node)
    		Service string
    		Bin     string
    	}
    }
    
  • Unable to decode hcl file data

    Unable to decode hcl file data

    I am using unmarshalwithconf to decode hcl file data. ( composing multiple decodehookFunc) But getting error to decode []map[string]interface{} data. Anyone can help ?

  • No pointer support

    No pointer support

    Hi,

    I'm trying to integrate cobra and koanf in a way that I can provide flags, env variables or an optional .env config file path in such a way that there is a hierarchy between those three input variants.

    I'm trying to have a struct and struct tags which map to (dot)env variables as well as to flag names. Pointer types in the struct are supposed to be somewhat optional but they seem to create some weird crash that I cannot exactly grasp.

    I have created a minimal example of the problem and a launch.json for vs code: https://github.com/jxsl13/koanf-test

    Starting this program instantly crashes. Maybe my approach with pointer values is incorrect and there might be a better approach.

  • Remove dependency on gopkg.in/yaml.v3

    Remove dependency on gopkg.in/yaml.v3

    Sadly, gopkg.in is offline or unreachable way too often, right now i get an error pulling refs from Github but it's a new one almost every week. For example

    https fetch: Get "https://gopkg.in/square/go-jose.v2?go-get=1": dial tcp 185.125.188.128:443: i/o timeout
    

    or

    github.com/knadh/koanf/parsers/yaml imports
        gopkg.in/yaml.v3 tested by
        gopkg.in/yaml.v3.test imports
        gopkg.in/check.v1: unrecognized import path "gopkg.in/check.v1": https fetch: Get "https://gopkg.in/check.v1?go-get=1": dial tcp 185.125.188.236:443: i/o timeout
    

    I suggest switching to goccy/go-yaml since it's a fork of the original package, is very well maintained and has the same API (plus nice formatted error messages). I am not affiliated with the project in any way, I'm just fed up with having to wait until gopkg.in fixes itself.

  • On PRs for new providers and parsers

    On PRs for new providers and parsers

    koanf currently bundles the most used providers and parsers. While only the providers that are actually used are compiled, new providers and parsers keep expanding the dependency tree that needs to be pulled and cached.

    Thus, it is not ideal to keep bundling more providers and parsers. Anyone who writes a provider or parser, we can list them in the docs for others to easily go get.

  • fatih/structs is not maintained anymore

    fatih/structs is not maintained anymore

    The maintainer of fatih/structs has stopped maintaining the package.

    I think it would be beneficial to replace this package with a maintained alternative.

    • https://github.com/fatih/structs
    • https://arslan.io/2018/10/09/taking-an-indefinite-sabbatical-from-my-projects/
🔥🔥 🌈 Golang configuration,use to Viper reading from remote Nacos config systems. Viper remote for Naocs.

Viper remote for Nacos Golang configuration,use to Viper reading from remote Nacos config systems. Viper remote for Naocs. runtime_viper := viper.New(

Dec 6, 2022
Tmpl - A tool to apply variables from cli, env, JSON/TOML/YAML files to templates

tmpl allows to apply variables from JSON/TOML/YAML files, environment variables or CLI arguments to template files using Golang text/template and functions from the Sprig project.

Nov 14, 2022
Golang Configuration tool that support YAML, JSON, TOML, Shell Environment

Configor Golang Configuration tool that support YAML, JSON, TOML, Shell Environment (Supports Go 1.10+) Usage package main import ( "fmt" "github.c

Dec 29, 2022
Generic templating tool with support of JSON, YAML and TOML data

gotempl Small binary used to generate files from Go Templates and data files. The following formats are supported: JSON YAML TOML Usage usage: gotempl

Jun 15, 2022
A Go (golang) environment loader (which loads env vars from a .env file)

A Go (golang) environment loader (which loads env vars from a .env file)

Feb 8, 2022
Library providing routines to merge and validate JSON, YAML and/or TOML files
Library providing routines to merge and validate JSON, YAML and/or TOML files

CONFLATE Library providing routines to merge and validate JSON, YAML, TOML files and/or structs (godoc) Typical use case: Make your application config

Sep 26, 2022
Jacket of spf13/viper: Simplified go configuration for wire-jacket.

Viper-Jacket: viper config for Wire-Jacket Jacket of spf13/viper: config for wire-jacket. Simplified env-based go configuration package using viper. b

Nov 18, 2021
Golang library for reading properties from configuration files in JSON and YAML format or from environment variables.

go-config Golang library for reading properties from configuration files in JSON and YAML format or from environment variables. Usage Create config in

Aug 22, 2022
JSON or YAML configuration wrapper with convenient access methods.

Config Package config provides convenient access methods to configuration stored as JSON or YAML. This is a fork of the original version. This version

Dec 16, 2022
It syncronizes the configuration described in a YAML file against your GitHub Organization

It syncronizes the configuration described in a YAML file against your GitHub Organization. Combined with a CI system, it can be used to implement GitOps for GitHub.

Jul 19, 2021
goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configuration file.

goconfig goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configur

Dec 15, 2022
Genv is a library for Go (golang) that makes it easy to read and use environment variables in your projects. It also allows environment variables to be loaded from the .env file.

genv Genv is a library for Go (golang) that makes it easy to read and use environment variables in your projects. It also allows environment variables

Dec 21, 2022
Go-yaml - Yaml parsing Toolkit For Golang

go-yaml 介绍 gopkg.in/yaml.v3 已经是个非常好用的包,但是在实际开发中总有类型转换带来的麻烦,go-yaml只是在它的基础上,简单的一层

Jan 13, 2022
Viper wrapper with config inheritance and key generation

piper - Simple Wrapper For Viper Single Source of Truth Generated Key Structs, No Typo Config Inheritance Multiple Config Strategies Support Cache For

Sep 26, 2022
formicidate is a small tool for Go application can update the value of environment variables in a .env file with code

formicidae Update .env files in Go with code. What is fomicidae? formicidate is a small tool for Go application. You can update the value of environme

Jan 23, 2022
YML2FSTAB - Convert from yml data to /etc/fstab configuration

YML2FSTAB - Convert from yml data to /etc/fstab configuration

Nov 1, 2021
Go library for the TOML language

go-toml Go library for the TOML format. This library supports TOML version v1.0.0-rc.3 Features Go-toml provides the following features for using data

Dec 27, 2022
TOML parser and encoder library for Golang

TOML parser and encoder library for Golang TOML parser and encoder library for Golang. This library is compatible with TOML version v0.4.0. Installati

Oct 11, 2022
shops is a simple command-line tool written in Go that helps you simplify the way you manage configuration across a set of machines.

shops is a simple command-line tool written in Go that helps you simplify the way you manage configuration across a set of machines. shops is your configuration management tool of choice when Chef, Puppet, Ansible are all too complicated and all you really want to do is run a bunch of regular shell against a set of hosts.

Jul 5, 2021