Harvest configuration, watch and notify subscriber

Harvester Running CI codecov Go Report Card GoDoc GitHub release

Harvester is a configuration library which helps setting up and monitoring configuration values in order to dynamically reconfigure your application.

Configuration can be obtained from the following sources:

  • Seed values, are hard-coded values into your configuration struct
  • Environment values, are obtained from the environment
  • Flag values, are obtained from CLI flags with the form -flag=value
  • File internals in local storage. Only text files are supported, don't use it for binary.
  • Consul, which is used to get initial values and to monitor them for changes

The order is applied as it is listed above. Consul seeder and monitor are optional and will be used only if Harvester is created with the above components.

Harvester expects a go structure with tags which defines one or more of the above like the following:

type Config struct {
    IndexName      sync.String          `seed:"customers-v1"`
    CacheRetention sync.Int64           `seed:"86400" env:"ENV_CACHE_RETENTION_SECONDS"`
    LogLevel       sync.String          `seed:"DEBUG" flag:"loglevel"`
    Signature      sync.String          `file:"signature.txt"`
    Sandbox        sync.Bool            `seed:"true" env:"ENV_SANDBOX" consul:"/config/sandbox-mode"`
    AccessToken    sync.Secret          `seed:"defaultaccesstoken" env:"ENV_ACCESS_TOKEN" consul:"/config/access-token"`
    WorkDuration   sync.TimeDuration    `seed:"1s" env:"ENV_WORK_DURATION" consul:"/config/work-duration"`
}

The above defines the following fields:

  • IndexName, which will be seeded with the value customers-v1
  • CacheRetention, which will be seeded with the value 18, and if exists, overridden with whatever value the env var ENV_CACHE_RETENTION_SECONDS holds
  • LogLevel, which will be seeded with the value DEBUG, and if exists, overridden with whatever value the flag loglevel holds
  • Sandbox, which will be seeded with the value true, and if exists, overridden with whatever value the env var ENV_SANDBOX holds and then from consul if the consul seeder and/or watcher are provided.
  • WorkDuration, which will be seeded with the value 1s, and if exists, overridden with whatever value the env var ENV_WORK_DURATION holds and then from consul if the consul seeder and/or watcher are provided.

The fields have to be one of the types that the sync package supports in order to allow concurrent read and write to the fields. The following types are supported:

  • sync.String, allows for concurrent string manipulation
  • sync.Int64, allows for concurrent int64 manipulation
  • sync.Float64, allows for concurrent float64 manipulation
  • sync.Bool, allows for concurrent bool manipulation
  • sync.Secret, allows for concurrent secret manipulation. Secrets can only be strings
  • sync.TimeDuration, allows for concurrent time.duration manipulation.

For sensitive configuration (passwords, tokens, etc.) that shouldn't be printed in log, you can use the Secret flavor of sync types. If one of these is selected, then at harvester log instead of the real value the text *** will be displayed.

Harvester has a seeding phase and an optional monitoring phase.

Seeding phase

  • Apply the seed tag value, if present
  • Apply the value contained in the env var, if present
  • Apply the value contained in the file, if present
  • Apply the value returned from Consul, if present and harvester is setup to seed from consul
  • Apply the value contained in the CLI flags, if present

Conditions where seeding fails:

  • If at the end of the seeding phase one or more fields have not been seeded
  • If the seed value is invalid

Seeder

Harvester allows the creation of custom getters which are used by the seeder and implement the following interface:

type Getter interface {
    Get(key string) (string, error)
}

Seed and env tags are supported by default, the Consul getter has to be setup when creating a Harvester with the builder.

Monitoring phase (Consul only)

  • Monitor a key and apply if tag key matches
  • Monitor a key-prefix and apply if tag key matches

Monitor

Harvester allows for dynamically changing the config value by monitoring a source. The following sources are available:

  • Consul, which supports monitoring for keys and key-prefixes.

This feature have to be setup when creating a Harvester with the builder.

Builder

The Harvester builder pattern is used to create a Harvester instance. The builder supports setting up:

  • Consul seed, for setting up seeding from Consul
  • Consul monitor, for setting up monitoring from Consul
    h, err := New(&cfg).
                WithConsulSeed("address", "dc", "token").
                WithConsulMonitor("address", "dc", "token").
                Create()

The above snippet set's up a Harvester instance with consul seed and monitor.

Notification support

In order to be able to monitor the changes in the configuration we provide a way to notify when a change is happening via the builder.

    h, err := harvester.New(&cfg).WithNotification(chNotify).Create()
    ...

Consul

Consul has support for versioning (ModifyIndex) which allows us to change the value only if the version is higher than the one currently.

Examples

Head over to examples readme on how to use harvester

How to Contribute

See Contribution Guidelines.

Code of conduct

Please note that this project is released with a Contributor Code of Conduct. By participating in this project and its community you agree to abide by those terms.

Comments
  • "CROSSSLOT Keys in request don't hash to the same slot" error

    Is your feature request related to a problem? Please describe

    In method getValues in monitor/redis/watcher.go:74 the method MGet cannot be used with an arbitrary set of keys as this leads to an error: CROSSSLOT Keys in request don't hash to the same slot, which means that MGet can only be used for set of keys that hash to the same slot.

    Is this a bug? Please provide steps to reproduce, a failing test etc.

    Describe the solution

    Additional context

  • Redis monitoring always setting the version to 0

    Redis monitoring always setting the version to 0

    Is your feature request related to a problem? Please describe

    When monitoring a Redis key, the version is always set to 0. See here:

    // file: /monitor/redis/watcher.go#L80
    
    changes = append(changes, change.New(config.SourceRedis, key, values[i].(string), 0))
    

    Then, when the field's value is set, it will always trigger a debug log and send a change notification, see here:

    // file: /config/config.go#L102-L123
    
    func (f *Field) Set(value string, version uint64) error {
    	if version != 0 && version < f.version { // <--- version is 0, so it will not match
    		log.Errorf("version %d is older than the field's %q (version %d)", version, f.name, f.version)
    		return nil
    	}
    
    	if version != 0 && version == f.version { // <--- version is 0, so it will not match
    		log.Debugf("version %d is the same as field's %q", version, f.name)
    		return nil
    	}
    
    	prevValue := f.structField.String()
    
    	if err := f.structField.SetString(value); err != nil {
    		return err
    	}
    
    	f.version = version
    	log.Debugf("field %q updated with value %q, version: %d", f.name, f, version) // <--- this one will always evaluate
    	f.sendNotification(prevValue, value) // <--- this one as well
    	return nil
    }
    

    2 issues:

    1. There will be one log for each Redis key and at each monitoring tick, even though the value didn't change
    2. It sends a notification to chNotifications for no reason

    Is this a bug? Please provide steps to reproduce, a failing test etc.

    Yes.

    To reproduce, monitor a Redis key, enable debug logs and tail the logs.

    Describe the solution

    We need to increment the Redis version if there is a change.

    Proposed solution:

    1. When reading a value from Redis, generate a checksum
    2. If the checksum is the same as the previously seen checksum for this key, discard the value (no changes)
    3. If the checksum is different, it means that there's a change. In this case, get the latest known version for this key, increment it and trigger a notification.
  • Changing Watcher interface for avoiding race-conditions

    Changing Watcher interface for avoiding race-conditions

    Is your feature request related to a problem? Please describe

    currently we have the following interface:

    type Watcher interface {
    	Watch(ctx context.Context, ch chan<- []*change.Change) error
    }
    

    This interface is a smell because it opens the possibility for someone different than the data-producer to close the channel - this situation is dangerous because it can lead to:

    1. race-conditions
    2. panics in case the Watcher tries to watch a closed channel

    I suggest to adapt the interface as:

    type Watcher interface {
    	Watch(ctx context.Context)  <-chan []*change.Change, error
    }
    

    Is this a bug? Please provide steps to reproduce, a failing test etc.

    Describe the solution

    Additional context

  • Add feature for seeding from files

    Add feature for seeding from files

    Which problem is this PR solving?

    Sometimes it is needed to seed some complex data from external file, and it is not fit well in ENV variables.

    Short description of the changes

    Add file seeding type. When specified, file is being read and populated in the corresponding field. Behaves the same way as other seeding types. No breaking changes introduced.

  • Add support for user defined custom types

    Add support for user defined custom types

    Is your feature request related to a problem? Please describe

    Perhaps it could be a good idea to give library users an ability to define and use their own custom field types, not limiting only to predefined sync package.

    It could be used for example to:

    1. Add any kind of custom seeding stage validation
    2. Make complex value types (for example, config string "[email protected]" could be translated at user defined struct {Name:”user”, Host:”example.com”} automatically on seeding stage

    Describe the solution

    Core idea - rely on interface, not on reflection based setter/stringer methods.

    1. Decouple config.Value and sync package types by moving the responsibility to set a string typed value to the sync types themselves. It could be done introducing something like SetString(string) error inside each type.
    2. Replace reflection based setter and printer fields of the sync.Field with CfgField interface
    3. Require that any new custom type structure must implement the CfgField interface and be thread safe
    4. Change behaviour on nesting config structs:
      1. If struct field has seeding tags, it MUST implement CfgField interface and it is tracked as a source for config.Field.
      2. If there are not seeding tags, this struct considered as a nested structure

    CfgField interface could look like:

    type CfgField interface {
    	//golang stringer interface
    	fmt.Stringer
    	//setter from string value, should parse string 
    	//to the underlying type and set a value, or return an error
    	SetString(value string) error
    }
    
  • Examples, basic, consul seed and monitoring

    Examples, basic, consul seed and monitoring

    Signed-off-by: Dimitris Baltas [email protected]

    Which problem is this PR solving?

    Examples for usage closes #19

    Short description of the changes

    Add example for

    • usage with ENV vars
    • usage with consul
  • Set harvester log level

    Set harvester log level

    Is your feature request related to a problem? Please describe

    I want to set the log level of your logging to a non-DEBUG value.

    Is this a bug? Please provide steps to reproduce, a failing test etc.

    Run harvester.Harvest for anything

    Describe the solution

    A way to set log level.

    Additional context

  • add support for time.duration

    add support for time.duration

    Which problem is this PR solving?

    Closes #64. Allow usage of time.Duration as field

    Short description of the changes

    Mainly adding the time.Duration as cfg type implemented interface.

  • Update consul

    Update consul

    Which problem is this PR solving?

    Bump consul-api lib to newest version in hope that it fixes the request timeout in watch

    Short description of the changes

    • Changed consul api to newest version
    • Changed docker-compose consul to 1.8.0
  • Add support for Debugf

    Add support for Debugf

    Which problem is this PR solving?

    Some logging could perhaps be reported as DEBUG level; this PR adds such method and uses it in 1 instance.

    There might be more places when this could be used.

    Short description of the changes

    Added a logging method Debugf

  • Vault support

    Vault support

    Is your feature request related to a problem? Please describe

    Secrets management is very important for security. Reading secrets from Vault will solve this problem. We need support for Vault in the seeing phase.

    Describe the solution

    • Add struct tags for Vault
    • Add seed support for Vault
  • Introduce functional options instead of Builder pattern

    Introduce functional options instead of Builder pattern

    Is your feature request related to a problem? Please describe

    We would like to use the more idiomatic functional options instead of the builder pattern. Ideally doing that in a non-breaking change with a deprecation notice that we will not support the old builder pattern anymore.

  • Support slice type values

    Support slice type values

    Is this a bug? Please provide steps to reproduce, a failing test etc.

    Add support to load slice field types - allowing to put slice type values in consul as lists.

    Describe the solution

    load the slice by the slice type (which must be one of the supported types implementing cfg type).

goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configuration file.

goconfig goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configur

Dec 15, 2022
✨Clean and minimalistic environment configuration reader for Golang

Clean Env Minimalistic configuration reader Overview This is a simple configuration reading tool. It just does the following: reads and parses configu

Jan 8, 2023
An opinionated configuration loading framework for Containerized and Cloud-Native applications.
An opinionated configuration loading framework for Containerized and Cloud-Native applications.

Opinionated configuration loading framework for Containerized and 12-Factor compliant applications. Read configurations from Environment Variables, an

Dec 16, 2022
go-up! A simple configuration library with recursive placeholders resolution and no magic.

go-up! A simple configuration library with placeholders resolution and no magic. go-up provides a simple way to configure an application from multiple

Nov 23, 2022
🛠 A configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP
🛠 A configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP

config A small configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP. Example func main() {

Dec 11, 2022
Manage local application configuration files using templates and data from etcd or consul

confd confd is a lightweight configuration management tool focused on: keeping local configuration files up-to-date using data stored in etcd, consul,

Dec 27, 2022
A flexible and composable configuration library for Go that doesn't suck

croconf A flexible and composable configuration library for Go that doesn't suck Ned's spec for Go configuration which doesn't suck: Fully testable: t

Nov 17, 2022
A flexible and composable configuration library for Go that doesn't suck

croconf A flexible and composable configuration library for Go Why? We know that there are plenty of other Go configuration and CLI libraries out ther

Nov 17, 2022
Golang library for reading properties from configuration files in JSON and YAML format or from environment variables.

go-config Golang library for reading properties from configuration files in JSON and YAML format or from environment variables. Usage Create config in

Aug 22, 2022
Jul 4, 2022
12 factor configuration as a typesafe struct in as little as two function calls

Config Manage your application config as a typesafe struct in as little as two function calls. type MyConfig struct { DatabaseUrl string `config:"DAT

Dec 13, 2022
JSON or YAML configuration wrapper with convenient access methods.

Config Package config provides convenient access methods to configuration stored as JSON or YAML. This is a fork of the original version. This version

Dec 16, 2022
Configure is a Go package that gives you easy configuration of your project through redundancy

Configure Configure is a Go package that gives you easy configuration of your project through redundancy. It has an API inspired by negroni and the fl

Sep 26, 2022
Load configuration in cascade from multiple backends into a struct
Load configuration in cascade from multiple backends into a struct

Confita is a library that loads configuration from multiple backends and stores it in a struct. Supported backends Environment variables JSON files Ya

Jan 1, 2023
Small library to read your configuration from environment variables

envconfig envconfig is a library which allows you to parse your configuration from environment variables and fill an arbitrary struct. See the example

Nov 3, 2022
A minimalist Go configuration library
A minimalist Go configuration library

fig fig is a tiny library for loading an application's config file and its environment into a Go struct. Individual fields can have default values def

Dec 23, 2022
Go configuration made easy!

gofigure Go configuration made easy! Just define a struct and call Gofigure Supports strings, ints/uints/floats, slices and nested structs Supports en

Sep 26, 2022
go implementation of lightbend's HOCON configuration library https://github.com/lightbend/config

HOCON (Human-Optimized Config Object Notation) Configuration library for working with the Lightbend's HOCON format. HOCON is a human-friendly JSON sup

Dec 3, 2022
Golang library for managing configuration data from environment variables

envconfig import "github.com/kelseyhightower/envconfig" Documentation See godoc Usage Set some environment variables: export MYAPP_DEBUG=false export

Dec 26, 2022