Load configuration in cascade from multiple backends into a struct

Build Status GoDoc Go Report Card

Confita is a library that loads configuration from multiple backends and stores it in a struct.

Supported backends

Install

go get -u github.com/heetch/confita

Usage

Confita scans a struct for config tags and calls all the backends one after another until the key is found. The value is then converted into the type of the field.

Struct layout

Go primitives are supported:

type Config struct {
  Host        string        `config:"host"`
  Port        uint32        `config:"port"`
  Timeout     time.Duration `config:"timeout"`
}

By default, all fields are optional. With the required option, if a key is not found then Confita will return an error.

type Config struct {
  Addr        string        `config:"addr,required"`
  Timeout     time.Duration `config:"timeout"`
}

Nested structs are supported too:

type Config struct {
  Host        string        `config:"host"`
  Port        uint32        `config:"port"`
  Timeout time.Duration     `config:"timeout"`
  Database struct {
    URI string              `config:"database-uri,required"`
  }
}

If a field is a slice, Confita will automatically split the config value by commas and fill the slice with each sub value.

type Config struct {
  Endpoints []string `config:"endpoints"`
}

As a special case, if the field tag is "-", the field is always omitted. This is useful if you want to populate this field on your own.

type Config struct {
  // Field is ignored by this package.
  Field float64 `config:"-"`

  // Confita scans any structure recursively, the "-" value prevents that.
  Client http.Client `config:"-"`
}

Loading configuration

Creating a loader:

loader := confita.NewLoader()

By default, a Confita loader loads all the keys from the environment. A loader can take other configured backends as parameters.

loader := confita.NewLoader(
  env.NewBackend(),
  file.NewBackend("/path/to/config.json"),
  file.NewBackend("/path/to/config.yaml"),
  flags.NewBackend(),
  etcd.NewBackend(etcdClientv3),
  consul.NewBackend(consulClient),
  vault.NewBackend(vaultClient),
)

Loading configuration:

err := loader.Load(context.Background(), &cfg)

Since loading configuration can take time when used with multiple remote backends, context can be used for timeout and cancelation:

ctx, cancel := context.WithTimeout(context.Background(), 5 * time.Second)
defer cancel()
err := loader.Load(ctx, &cfg)

Default values

If a key is not found, Confita won't change the respective struct field. With that in mind, default values can simply be implemented by filling the structure before passing it to Confita.

type Config struct {
  Host        string        `config:"host"`
  Port        uint32        `config:"port"`
  Timeout     time.Duration `config:"timeout"`
  Password    string        `config:"password,required"`
}

// default values
cfg := Config{
  Host: "127.0.0.1",
  Port: "5656",
  Timeout: 5 * time.Second,
}

err := confita.NewLoader().Load(context.Background(), &cfg)

Backend option

By default, Confita queries each backend one after another until a key is found. However, in order to avoid some useless processing the backend option can be specified to describe in which backend this key is expected to be found. This is especially useful when the location of the key is known beforehand.

type Config struct {
  Host        string        `config:"host,backend=env"`
  Port        uint32        `config:"port,required,backend=etcd"`
  Timeout     time.Duration `config:"timeout"`
}

Command line flags

The flags backend allows to load individual configuration keys from the command line. The default values are extracted from the struct fields values.

A short option is also supported.

To update usage message on the command line, provide a description to the given field.

type Config struct {
  Host        string        `config:"host,short=h"`
  Port        uint32        `config:"port,short=p"`
  Timeout     time.Duration `config:"timeout,description=timeout (in seconds) for failure"`
}
./bin -h

Usage of ./bin:
  -host string
       (default "127.0.0.1")
  -h string
       (default "127.0.0.1")
  -port int
       (default 5656)
  -p int
       (default 5656)
  -timeout duration
       timeout (in seconds) for failure (default 10s)

License

The library is released under the MIT license. See LICENSE file.

Owner
Heetch
Enjoy your night out
Heetch
Comments
  • struct backend

    struct backend

    this is my attempt at a fix for issue #24

    The work isn't finished yet (except for int, string and bool types) but I wanted to open the PR earlier to get some feedback on whether there is a quicker way of doing this, as I feel like I'm doing a lot of manual work.

  • Map backend #12 - added a map to store default in backend

    Map backend #12 - added a map to store default in backend

    In this pull request, we are adding a map of defaults for usage by the different backends. Based on the ticket, I believe this is what is required, however I do apologize if I misunderstood it

    I also wasn't sure if a the type map[string][string] is the best solution. I was thinking of using a map[string]interface{}, but wasn't sure if the user would want to define an object as a value

    Issue # 12

  • adds ssm backend

    adds ssm backend

    Adds Amazon SSM's Parameters as a backend.

    There are many ways the calls could be made to Amazon, but, just starting with the simplest now. User just gives the path for their parameter store. Can be easily changed in the future if people want to use it in a different way. Very open to ideas here.

  • When value defined in multiple backend, last one wins

    When value defined in multiple backend, last one wins

    This was previously discussed in https://github.com/heetch/confita/issues/7, but my tests are giving different results, If I define a variable in a file as an env var, the last backend on the list wins

    Examples: l := confita.NewLoader(file.NewBackend(path),env.NewBackend()) in this case the env backend overwrites the values in the file files

    n := confita.NewLoader(env.NewBackend(),file.NewBackend(path)) same issue here, the file will overwrite any value previously set as env var

    The docs say: "Confita scans a struct for config tags and calls all the backends one after another until the key is found" from my understanding once found it stops and continue looking for the next config key

    There is a working example of the issue:

    https://play.golang.org/p/c8l9NbRnJOT

    Thanks to @rogpeppe for your go playground example on issue 61

  • Support short/long flags

    Support short/long flags

    Most command-line tools support a short and a long form for flags, e.g. "-p" or "--port". Currently confita only supports a single flag per member.

    Suggestion: add a "short" sub-tag and handle it as a synonym of the default name. Also update help usage generation to reflect this.

    type Configuration struct {
    	Port          int    `config:"port,short=p"`
    }
    
    
  • parse yaml file failled, I have that key but confita said not found

    parse yaml file failled, I have that key but confita said not found

    hi, guys. I met some trouble with use confita. Anybody can help me?Please. I have a nest struct:

    type Database struct {
    	Which 	string  `config:"which"`
    	DbDir 	string	`config:"db_dir"`
    }
    
    type Config struct {
    	Database Database 	`config:"database,required"`
    	AppName string 		`config:"app_name,required"`
    }
    

    and then,when i use goconvey to test, it failed. here is my code:

    func createTempFile(name, content string) (string, func()) {
    	dir, err := ioutil.TempDir("", "testDir")
    	So(err, ShouldBeNil)
    
    	path := filepath.Join(dir, name)
    	f, err := os.Create(path)
    	So(err, ShouldBeNil)
    
    	_, err = fmt.Fprint(f, content)
    	So(err, ShouldBeNil)
    	So(f.Close(), ShouldBeNil)
    
    	return path, func() {
    		So(os.RemoveAll(dir), ShouldBeNil)
    	}
    }
    
    func TestLoadWithTempFile(t *testing.T) {
    	Convey("test Load with temp file", t, func() {
    		var cfg Config
    		path, cleanUp := createTempFile("app.yaml", `
    app_name: blog
    database:
      which: sqlite3
      db_dir: /tmp/test.db
    `)
    		defer cleanUp()
    
    		loader := confita.NewLoader(file.NewBackend(path))
    		err := loader.Load(context.Background(), &cfg)
    
    		So(err, ShouldBeNil)
    		So(cfg.Database, ShouldNotBeNil)
    		So(cfg.AppName, ShouldEqual, "blog")
    		So(cfg.Database.Which, ShouldEqual, "sqlite3")
    		So(cfg.Database.DbDir, ShouldEqual, "/tmp/test.db")
    	})
    }
    

    and the test result:

    ....x.
    Failures:
    
      * /home/lks/go/src/github.com/crazypandas/goBlog/config/config_test.go 
      Line 46:
      Expected: nil
      Actual:   'required key 'app_name' for field 'AppName' not found'
    
    
    6 total assertions
    

    Is there anything wrong?

  • Issue when trying to load a list of struct from a yaml file.

    Issue when trying to load a list of struct from a yaml file.

    Given this type:

    type migrationSettings struct {
    	Source      string `config:"source,required"`
    	Destination string `config:"destination,required"`
    	Partitions  int    `config:"partitions,required"`
    }
    

    The following yaml:

    - source: topic1
      destination: topic2
      partitions: 3
    
    - source: topic1
      destination: topic2
      partitions: 3
    
    - source: topic1
      destination: topic2
      partitions: 3
    

    The snippet:

    var s []migrationSettings
    
    l := confita.NewLoader(file.NewBackend("config.yaml"))
    l.Load(context.Background(), &s)
    
    fmt.Println(s)
    

    The output:

    []
    
  • Go modules

    Go modules

    Fix #40

    Hi @YassineE :-) Hope your are doing well. Recently I am trying to get more and more familiar with the modules in the Go 1.11. Hence this PR is a great opportunity to apply this approach to the real project.

    Few notes about the PR:

    1. My source of knowledge how to introduce go modules and configure travis build are following articles:
    1. Correct me if I am wrong, but since repository is an open source library you want to still have support for different go versions. Therefore we still have to keep go dep for go 1.9.x & 1.10.x + I've introduced go mod for go 1.11.x & tip. I've managed to configure a build matrix to satisfy these requirements.
    2. I've marked tip go version in the matrix as the one which can fail. It's ok if the code fails on unstable development versions of Go, however it should not break the whole build if other version are green.
    3. I've tried my best to avoid duplication in .travis.yml, unfortunately not everywhere it was possible. It is because we have 4 go versions and 2 different dependencies managment strategies. Let me know if you have any idea how to improve it.

    Waiting for the feedback!

  • Specify the backend

    Specify the backend

    In order to avoid some useless processing we can specify for a field in which backend we can find its value.

    e.g.

    type Config struct {
      Foo        string        `config:"foo",bcknd:"etcd"`
      Bar        string        `config:"bar",bcknd:"consul"`
    }
    
  • Environment variable specification

    Environment variable specification

    The XCU specification says that the environment variable names consist solely of upper-case letters, digits and the "_" (underscore).

    Maybe, we should respect this spec and not handle the case if names contains "-" (dash).

  • Improve documentation

    Improve documentation

    IMO we should specify in the doc that when a key is found in a backend, we won't continue to look for it in the remaining backends.

    So there is an importance on how the backends are sorted inside the slice.

    What do you think ?

  • Transition from dgrijalva/jwt-go pkg to golang-jwt/jwt

    Transition from dgrijalva/jwt-go pkg to golang-jwt/jwt

    jwt-go has a security vulnerability (CVE-2020-26160). There is no patch available and users of jwt-go are advised to migrate to golang-jwt at version 3.2.1. 👀

  • Reusing nested config structs

    Reusing nested config structs

    Hi! I am using confita configs as API for internal libraries in my projects. I've tried to use config structs multiple times in single project setup. And I found a problem for my usage. Field tags can't repeat in one config:

    type HTTPConfig struct {
    	Addr string `config:"addr"`
    	Port int `config:"port"`
    }
    
    type AppConfig struct {
    	MetricsHTTPServer HTTPConfig
    	LoggingHTTPServer HTTPConfig
    }
    

    There are not possibilities for defining different options for different servers. Also if we'll try to use flags as backend, we'll get panic (below):

    panic: ./cmd/example/example flag redefined: addr
    
    goroutine 1 [running]:
    flag.(*FlagSet).Var(0xc0001371a0, 0x551e910, 0xc000614440, 0x4f288db, 0xc, 0x0, 0x0)
    	/-S/contrib/go/_std/src/flag/flag.go:871 +0x485
    flag.(*FlagSet).StringVar(...)
    	/-S/contrib/go/_std/src/flag/flag.go:760
    flag.(*FlagSet).String(0xc0001371a0, 0x4f288db, 0xc, 0x0, 0x0, 0x0, 0x0, 0xc000614430)
    	/-S/contrib/go/_std/src/flag/flag.go:773 +0xa5
    flag.String(...)
    	/-S/contrib/go/_std/src/flag/flag.go:780
    github.com/heetch/confita/backend/flags.(*Backend).LoadStruct(0x5d76df8, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x0, 0x0)
    	/-S/vendor/github.com/heetch/confita/backend/flags/flags.go:73 +0x548
    github.com/heetch/confita.(*Loader).resolve(0xc00025f050, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x5d30240, 0x199)
    	/-S/vendor/github.com/heetch/confita/config.go:171 +0x788
    github.com/heetch/confita.(*Loader).Load(0xc00025f050, 0x552dea8, 0xc0005d2340, 0x4f63ac0, 0x5d30240, 0x49, 0xd)
    	/-S/vendor/github.com/heetch/confita/config.go:58 +0x20a
    

    It may be solved using something like prefixes. Example usage is following:

    type HTTPConfig struct {
    	Addr string `config:"addr"`
    	Port int `config:"port"`
    }
    
    type AppConfig struct {
    	MetricsHTTPServer HTTPConfig `config:"prefix=metrics"`
    	LoggingHTTPServer HTTPConfig `config:"prefix=logging"`
    }
    

    Or more idiomatic if it needs. I think this feature will be useful not only me))

  • Can't load config into struct with json tags

    Can't load config into struct with json tags

    I noticed a regression in behaviour. Here is code for reproducing the issue.

    package confitajson
    
    import (
    	"context"
    	"log"
    	"testing"
    
    	"github.com/heetch/confita"
    	"github.com/heetch/confita/backend/file"
    )
    
    func TestLoad(t *testing.T) {
    	b := file.NewBackend("config.json")
    
    	var c struct {
    		Field string `json:"field"`
    	}
    
    	if err := confita.NewLoader(b).Load(context.Background(), &c); err != nil {
    		log.Fatalf("failed to load config: %v", err)
    	}
    
    	if c.Field != "value" {
    		log.Fatal("field value is not updated")
    	}
    }
    

    Content of config.json file is following:

    {
        "field": "value"
    }
    

    When running this test on the latest confita release, struct field is not updated. But on version v0.7.0 code works fine.

    prime@bee ~/C/confitajson> go get github.com/heetch/[email protected]
    go: downloading github.com/heetch/confita v0.7.0
    prime@bee ~/C/confitajson> go test .
    ok  	github.com/slon/confitajson	0.002s
    prime@bee ~/C/confitajson> go get github.com/heetch/confita
    go: github.com/heetch/confita upgrade => v0.9.1
    prime@bee ~/C/confitajson> go test .
    2020/05/27 20:02:34 field value is not updated
    FAIL	github.com/slon/confitajson	0.002s
    FAIL
    

    I also noticed, that removing these 3 lines, seems to fix this issue for me. https://github.com/heetch/confita/blob/master/config.go#L183-L185

    Is that change in behaviour intentional, or is it indeed a regression?

  • Structured configuration support (path/dots)

    Structured configuration support (path/dots)

    Does confita support structured configuration? It would be useful to make configurations more clean/organized/structure, and for configurations of growing/large projects.

    I have read the README.md, but there's no example for structure config,... And, I didn't find any TOML/YAML/JSON examples of structured configurations.

    I would like to populate the following structure:

    
    type Config struct {
    	// notifications configuration
    	Notifications struct {
    		// turns on user notifications for changes/updates/creations
    		Enabled bool
    		// from email address of emails
    		From string
    		// BCC addresses of every notification sent
    		BCC []string
    	}
    
    	// SMTP server/mail-sending configuration
    	SMTP struct {
    		// format server:port
    		ServerAddr string
    
    		// plain auth credentials
    		Username, Password string
    
    		// default email address to sent from
    		DefaultFrom string
    	}
    }
    

    For example, with TOML:

    notifications.enabled = true
    notifications.from = "[email protected]"
    notifications.bcc = "[email protected]"
    
    smtp.serveraddr = "smtp.gmail.com:587"
    smtp.username = "[email protected]"
    smtp.password = "secret"
    smtp.defaultfrom = "[email protected]"
    

    Or, with YAML:

    notifications:
      - enabled: true
      - from: "[email protected]"
      - bcc:  "[email protected]"
    
    smtp:
      - serveraddr: "smtp.gmail.com:587"
      - username: "[email protected]"
      - password: "secret"
      - defaultfrom: "[email protected]"
    
Load configuration from multiple sources in Go

Multiconfig Load configuration from multiple sources. Multiconfig makes loading/parsing from different configuration sources an easy task. The problem

Dec 2, 2022
Jul 4, 2022
12 factor configuration as a typesafe struct in as little as two function calls

Config Manage your application config as a typesafe struct in as little as two function calls. type MyConfig struct { DatabaseUrl string `config:"DAT

Dec 13, 2022
Tis module used as base fo configuration apps.By default, it expands into the inside of the application.

Tis module used as base fo configuration apps.By default, it expands into the inside of the application. Also, module c reads a dictionary of secrets from the application directory by its AppName and extension json.

Dec 7, 2021
⚙️ Dead Simple Config Management, load and persist config without having to think about where and how.

Configo Dead Simple Config Management, load and persist config without having to think about where and how. Install go get github.com/UltiRequiem/conf

Apr 6, 2022
Quick and easy way to load config files based on a simple set of rules.
Quick and easy way to load config files based on a simple set of rules.

config Quick and easy way to load config files based on a simple set of rules. Project inspired by https://github.com/lorenwest/node-config Important

Apr 9, 2021
A Go library for parsing struct tags from environment variables.

Envconfig Envconfig populates struct field values based on environment variables or arbitrary lookup functions. It supports pre-setting mutations, whi

Jan 2, 2023
✨Clean and minimalistic environment configuration reader for Golang

Clean Env Minimalistic configuration reader Overview This is a simple configuration reading tool. It just does the following: reads and parses configu

Jan 8, 2023
JSON or YAML configuration wrapper with convenient access methods.

Config Package config provides convenient access methods to configuration stored as JSON or YAML. This is a fork of the original version. This version

Dec 16, 2022
Configure is a Go package that gives you easy configuration of your project through redundancy

Configure Configure is a Go package that gives you easy configuration of your project through redundancy. It has an API inspired by negroni and the fl

Sep 26, 2022
An opinionated configuration loading framework for Containerized and Cloud-Native applications.
An opinionated configuration loading framework for Containerized and Cloud-Native applications.

Opinionated configuration loading framework for Containerized and 12-Factor compliant applications. Read configurations from Environment Variables, an

Dec 16, 2022
Small library to read your configuration from environment variables

envconfig envconfig is a library which allows you to parse your configuration from environment variables and fill an arbitrary struct. See the example

Nov 3, 2022
A minimalist Go configuration library
A minimalist Go configuration library

fig fig is a tiny library for loading an application's config file and its environment into a Go struct. Individual fields can have default values def

Dec 23, 2022
go-up! A simple configuration library with recursive placeholders resolution and no magic.

go-up! A simple configuration library with placeholders resolution and no magic. go-up provides a simple way to configure an application from multiple

Nov 23, 2022
Go configuration made easy!

gofigure Go configuration made easy! Just define a struct and call Gofigure Supports strings, ints/uints/floats, slices and nested structs Supports en

Sep 26, 2022
Harvest configuration, watch and notify subscriber

Harvester Harvester is a configuration library which helps setting up and monitoring configuration values in order to dynamically reconfigure your app

Dec 26, 2022
go implementation of lightbend's HOCON configuration library https://github.com/lightbend/config

HOCON (Human-Optimized Config Object Notation) Configuration library for working with the Lightbend's HOCON format. HOCON is a human-friendly JSON sup

Dec 3, 2022
🛠 A configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP
🛠 A configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP

config A small configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP. Example func main() {

Dec 11, 2022