Pit-of-vipers - Managing many viper instances

A pit of vipers

Objective

Do you use golang? Do you use viper for configuration management within golang? Are you sick and tired of:

  • Only handling one config file at a time

  • Being thread-unsafe

  • Having to write dodgy synchronization code to merge multiple config files from multiple source deterministically

  • Never going through the effort of live-reloading config on filesystem updates

Look no more! Pit of Vipers ingests as many viper instances as you want and:

  • Deterministically merges them in the order provided

  • Updates the merged config every time a sub-instance receives an update from the filesystem

  • Forces use of live-reloaded config

  • Does all of this in a thread-safe manner

How to

TL;DR:

package main

import (
	"fmt"
	viperpit "github.com/ajpauwels/pit-of-vipers"
)

type Config struct {
	Host      string `mapstructure:"host"`
	Port      uint16 `mapstructure:"port"`
	SecretKey string `mapstructure:"secretkey"`
	NewKey    string `mapstructure:"newkey"`
}

func main() {
	vpCh, errCh := viperpit.NewFromPaths([]string{"./config", "./config/shared", "./config/app", "./config/preview/shared", "./config/preview/app"}) // (1)
	for { // (2)
		select {
		case vp := <-vpCh: // (3)
			var config Config
			vp.Unmarshal(&config)
			fmt.Printf("%+v\n", config)
		case err := <-errCh: // (4)
			fmt.Errorf("%s", err)
		}
	}
}

Explanation of highlighted portions below:

  1. Calling a New* function on viperpit returns two channels: the viper channel which receives a merged viper instance every time one of the sub-instances is updated, and an error channel which receives all errors which occurred in this process

  2. Main thread just loops infinitely on a channel select statement, waiting for config updates or errors in the config update process

  3. The <-vpCh case receives a fully merged viper instance every time one of the sub-instances is updated from the filesystem

  4. The <-errCh case receives any errors that may have occurred during the merging process

Comments
  • Added NewFromPathsAndGlob to read multiple files from dirs

    Added NewFromPathsAndGlob to read multiple files from dirs

    Hi,

    Thanks for this useful library. I added a new initialization function that is especially suited for reading config files from a conf.d directory, i.e. multiple files with the same suffix.

    Cheers, Shmul

  • Does it merge map as well?

    Does it merge map as well?

    config.yaml

    database:
      uri: my-uri
    

    secret.yaml

    database:
      password: my-secret
    

    vpCh, errCh := viperpit.NewFromPaths([]string{"./"}) is not merging the configs from the root.

  • Use defaults

    Use defaults

    Hi,

    I noticed that the current implementation doesn't use default values set by viper.SetDefault, as a new viper instance is used for base rather than the default/static one. I therefore added an option to control which instance (if any) is used for the base. The implementation uses the Options Pattern which doesn't break the existing API, so was easy to add.

    Cheers, Shmul

  • feat: add support for remote config watching

    feat: add support for remote config watching

    Adds support for watching remote configs and updated them the same way as files on the filesystem. Uses polling. This PR does not break existing interface but adds a parallel constructor.

  • question: does/could this support viper remote with consul

    question: does/could this support viper remote with consul

    could this also support viper instances that also use github.com/spf13/viper/remote which calls the AddRemoteProvider() function ? e.g. https://madflojo.medium.com/using-viper-with-consul-to-configure-golang-applications-eaa84394b8de

Instances-api - A JSON API to fetch Piped instances

Instances-Api Hosting it yourself docker-compose up -d You can now access the AP

Dec 27, 2022
CraftTalk Command Line Tool helps with managing CraftTalk releases on baremetal instances

ctcli - CraftTalk Command Line Tool Commands help Shows help version Shows version init Initializes specified root directory as a ctcli dir. ctcli --r

Jan 20, 2022
The dynamic infrastructure framework for everybody! Distribute the workload of many different scanning tools with ease, including nmap, ffuf, masscan, nuclei, meg and many more!
The dynamic infrastructure framework for everybody! Distribute the workload of many different scanning tools with ease, including nmap, ffuf, masscan, nuclei, meg and many more!

Axiom is a dynamic infrastructure framework to efficiently work with multi-cloud environments, build and deploy repeatable infrastructure focussed on

Dec 30, 2022
Count once - Just once? no, when appear many it run once, but it can run many times

countOnce just once? no, when appear many it run once, but it can run many times

Jan 29, 2022
Light weight, extensible configuration management library for Go. Built in support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.
Light weight, extensible configuration management library for Go. Built in support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.

koanf (pronounced conf; a play on the Japanese Koan) is a library for reading configuration from different sources in different formats in Go applicat

Jan 8, 2023
Opinionated Go starter with gin for REST API, logrus for logging, viper for config with added graceful shutdown

go-gin-starter An opinionated starter for Go Backend projects using: gin-gonic/gin as the REST framework logrus for logging viper for configs Docker f

Dec 2, 2022
Jacket of spf13/viper: Simplified go configuration for wire-jacket.

Viper-Jacket: viper config for Wire-Jacket Jacket of spf13/viper: config for wire-jacket. Simplified env-based go configuration package using viper. b

Nov 18, 2021
Viper wrapper with config inheritance and key generation

piper - Simple Wrapper For Viper Single Source of Truth Generated Key Structs, No Typo Config Inheritance Multiple Config Strategies Support Cache For

Sep 26, 2022
Viper: a complete configuration solution for Go applications including 12-Factor apps
Viper: a complete configuration solution for Go applications including 12-Factor apps

Viper v2 feedback Viper is heading towards v2 and we would love to hear what you would like to see in it. Share your thoughts here: https://forms.gle/

Dec 6, 2021
Go starter project with Gin, Viper, postgres , redis, zap, prometheus metrics etc setup

Go REST Service Starter/Boilerplate Easily extendible REST API Service boilerplate aiming to follow idiomatic go and best practice. Any feedback and p

Jun 23, 2022
This Go based project of Aadhyarupam Innovators demonstrate the code examples for building microservices, integration with cloud services (Google Cloud Firestore), application configuration management (Viper) etc.

This Go based project of Aadhyarupam Innovators demonstrate the code examples for building microservices, integration with cloud services (Google Cloud Firestore), application configuration management (Viper) etc.

Dec 22, 2022
Api-go-gin-viper - Simple representaion on how to implement CRUD functionality in API using Go programming language

Simple API implementaion in Go To Get Started Clone repo Run the command to clon

Feb 18, 2022
Composable chains of nested http.Handler instances.

chain go get github.com/codemodus/chain Package chain aids the composition of nested http.Handler instances. Nesting functions is a simple concept. I

Sep 27, 2022
Get cloud instances with your favourite software pre-loaded

This Golang package can be used to provision cloud hosts using a simple CRUD-style API along with a cloud-init user-data script. It could be used to automate anything from k3s clusters, to blogs, or CI runners. We use it to create the cheapest possible hosts in the cloud with a public IP address.

Dec 14, 2022
Kubegres is a Kubernetes operator allowing to create a cluster of PostgreSql instances and manage databases replication, failover and backup.

Kubegres is a Kubernetes operator allowing to deploy a cluster of PostgreSql pods with data replication enabled out-of-the box. It brings simplicity w

Dec 30, 2022
This plugin helps you to use the AWS Command Line Interface (AWS CLI) to start and end sessions to your managed instances

Session Manager Plugin This plugin helps you to use the AWS Command Line Interface (AWS CLI) to start and end sessions to your managed instances. Sess

Dec 28, 2022
go-awssh is a developer tool to make your SSH to AWS EC2 instances easy.

Describing Instances/VPCs data, select one or multiple instances, and make connection(s) to selected instances. Caching the response of API calls for 1day using Tmpfs.

Oct 11, 2021
support multi autosolve instances for AYCD

AYCD AutoSolve Go Client Client for connecting to the AYCD AutoSolve network

Jun 26, 2022
This tutorial walks you through using the Cloud Functions minimal instances feature to mitigate cold starts.
This tutorial walks you through using the Cloud Functions minimal instances feature to mitigate cold starts.

Tutorial This tutorial walks you through using the Cloud Functions minimal instances feature to mitigate cold starts. Let’s take a deeper look at min

Jun 1, 2022
A Grafana backend plugin for automatic synchronization of dashboard between multiple Grafana instances.

Grafana Dashboard Synchronization Backend Plugin A Grafana backend plugin for automatic synchronization of dashboard between multiple Grafana instance

Dec 23, 2022