a dynamic configuration framework used in distributed system

go-archaius

Build Status Coverage Status HitCount goproxy.cn

This is a light weight configuration management framework which helps to manage configurations in distributed system

The main objective of go archaius is to pull and sync the configuration from multiple sources

Why use go-archaius

it is hard to manage configurations in a distributed system. archaius is able to put all configuration in distributed system together and manage them. To make it simple to get the exact config you want in distributed system. It also keeps watching configuration changes, and fire change event if value changes. so that you can easily implement a service which has hot-reconfiguration features. when you need to change configurations, your service has zero-down time.

Conceptions

Sources

Go-archaius can manage multiple sources at the same time. Each source can holds same or different key value pairs. go-archaius keeps all the sources marked with their precedence, and merge key value based on precedence. in case if two sources have same key then key with higher precedence will be selected, and you can use archaius API to get its value

Here is the precedence list:

0: remote source - pull remote config server data into local

1: Memory source - after init, you can set key value in runtime.

2: Command Line source - read the command lines arguments, while starting the process.

3: Environment Variable source - read configuration in Environment variable.

4: Files source - read files content and convert it into key values based on the FileHandler you define

Dimension

It only works if you enable remote source, as remote server, it could has a lot of same key but value is different. so we use dimension to identify kv. you can also get kv in other dimension by add new dimension

Event management

You can register event listener by key(exactly match or pattern match) to watch value change.

File Handler

It works in File source, it decide how to convert your file to key value pairs. check FileHandler, currently we have 2 file handler implementation

archaius API

developer usually only use API to interact with archaius, check API.

To init archaius

archaius.Init()

when you init archaius you can decide what kind of source should be enable, required file slice was given, archaius checks file existing and add them into file source, if not exist, init fails, below example also enables env and mem sources.

	err := archaius.Init(
		archaius.WithRequiredFiles([]string{filename1}),
		archaius.WithOptionalFiles([]string{filename2}),
		archaius.WithENVSource(),
		archaius.WithMemorySource())

Put value into archaius

Notice, key value will be only put into memory source, it could be overwritten by remote config as the precedence list

archaius.Set("interval", 30)
archaius.Set("ttl", "30s")
archaius.Set("enable", false)

Read config files

if you have a yaml config

some:
  config: 1
ttl: 30s
service:
  name: ${NAME||go-archaius}
  addr: ${IP||127.0.0.1}:${PORT||80} 

after adding file

archaius.AddFile("/etc/component/xxx.yaml")

you can get value

ttl := archaius.GetString("ttl", "60s")
i := archaius.GetInt("some.config", "")
serviceName := archaius.GetString("service.name", "")
serviceAddr := archaius.GetString("service.addr", "")

note:

  1. For service.name config with value of ${NAME||go-archaius} is support env syntax. If environment variable ${NAME} isn't setting, return default value go-archaius. It's setted, will get real environment variable value. Besides, if ${Name^^} is used instead of ${Name}, the value of environment variable Name will be shown in upper case, and ${Name,,} will bring the value in lower case.
  2. For service.addr config is support "expand syntax". If environment variable ${IP} or ${PORT} is setted, will get env config. eg: export IP=0.0.0.0 PORT=443 , archaius.GetString("service.addr", "") will return 0.0.0.0:443 .

if you want to read some.config from env you can run

export some_config=xxxx

then you can read it by below code

i := archaius.GetInt("some.config", "")

by default archaius only support yaml files, but you can extend file handler to handle file in other format, for example we only consider file name as a key, content is the value.

archaius.AddFile("xxx.txt", archaius.WithFileHandler(util.FileHandler(util.UseFileNameAsKeyContentAsValue))

you can get value

v := archaius.GetString("/etc/component/xxx.txt", "")

Enable remote source

If you want to use one remote source, you must import the corresponding package of the source in your code.

import _ "github.com/go-chassis/go-archaius/source/remote/kie"

set remote info to init remote source

	ri := &archaius.RemoteInfo{
	//input your remote source config
	}
	//manage local and remote key value at same time
	err := archaius.Init(
		archaius.WithRequiredFiles([]string{filename1}),
		archaius.WithOptionalFiles([]string{filename2}),
		archaius.WithRemoteSource(archaius.KieSource, ri),
	)

Supported distributed configuration management service:

name import description
kie github.com/go-chassis/go-archaius/source/remote/kie ServiceComb-Kie is a config server which manage configurations in a distributed system. It is also a micro service in ServiceComb ecosystem and developed by go-chassis we call it ServiceComb Native application. https://kie.readthedocs.io
config-center github.com/go-chassis/go-archaius/source/remote/configcenter huawei cloud CSE config center https://www.huaweicloud.com/product/cse.html
apollo github.com/go-chassis/go-archaius/source/apollo A reliable configuration management system https://github.com/ctripcorp/apollo

Example: Manage local configurations

Complete example

Example: Manage key value change events

Complete example

Example: Manage remote source configurations

Complete example

Example: Manage module change events

Sometimes, we may want to handle multiple key value changes as a module, which means that the different key in the module has some relation with each other. Once such keys have changed, we expect to handle the changes as a whole instead of one by one. Module events help us to handle this case.

Complete example

Owner
focusing on go cloud native framework and service mesh solution to help developer to develop cloud service easily
null
Comments
  • 配置热更新回调接口修改成同一个module里的event就通知一次。

    配置热更新回调接口修改成同一个module里的event就通知一次。

    场景:

    项目中配置的使用,一般都是多个配置项合在一起使用,比如mysql的配置一般都会有账号、密码、端口地址等。如果在项目运行过程中,需要修改该项配置,同时修改了账号、密码和端口。event就会通知同一个模块多次,业务方没有考虑这种情况的时候,就会热更新多次,存在问题,即使订阅多个key,感觉也不能避免回调方回调多次的情况。

    建议:

    是否可以考虑将Listener接口的event做成切片,这样各个配置源就有更大的操作空间,给予到业务方就会只有一个热更新通知。

    type Listener interface {
    	Event(event *Event)
    }
    

    修改为:

    type Listener interface {
    	Event(event []*Event)
    }
    

    接口地址:

    https://github.com/go-chassis/go-archaius/blob/f6cc7f3c52efe30aebfe7e5b51ebbf64959c08ad/event/event_system.go#L53

  • 对于配置文件的配置读取,新增环境变量语法的兼容。

    对于配置文件的配置读取,新增环境变量语法的兼容。

    issus地址:https://github.com/go-chassis/go-archaius/issues/108

    例如下面的例子,使用如果配置文件中的value有${GO||/usr/local/go}这种格式的字符串, 就去获取一下环境变量${GO}的值,如果环境变量为空,就返回||与)之间的默认字符串。

  • 实现自定义配置中心的Client interface 参数疑问

    实现自定义配置中心的Client interface 参数疑问

    文件名

    source/remote/client.go

    接口

    type Client interface {
    	//PullConfigs pull all configs from remote
    	PullConfigs(labels ...map[string]string) (map[string]interface{}, error)
    	//PullConfig pull one config from remote
    	PullConfig(key, contentType string, labels map[string]string) (interface{}, error)
    	// PushConfigs push config to cc
    	PushConfigs(data map[string]interface{}, labels map[string]string) (map[string]interface{}, error)
    	// DeleteConfigsByKeys delete config for cc by keys
    	DeleteConfigsByKeys(keys []string, labels map[string]string) (map[string]interface{}, error)
    	//Watch get kv change results, you can compare them with local kv cache and refresh local cache
    	Watch(f func(map[string]interface{}), errHandler func(err error), labels map[string]string) error
    	Options() Options
    }
    

    PullConfig

    这个参数:contentType是什么意思


    PS.目前我们的项目正在使用go-chassis来开发一个业务框架,已有的配置中心为apollo,正在做support apollo的工作,想问下目前为什么停止对apollo的开发了?

  • custom Event not work

    custom Event not work

    Hi, I Have a custom event listen for config file changes, but it not work, can you help me ?

    package main
    
    import (
    	"fmt"
    	"github.com/go-chassis/go-archaius"
    	"github.com/go-chassis/go-archaius/core"
    )
    type Listener struct {
    	Key string
    }
    func (e *Listener) Event(event *core.Event) {
    	//  I want to dynamically get the latest values here.
    	fmt.Printf("****************%v",event)
    }
    func main(){
    	archaius.Init(archaius.WithRequiredFiles([]string{
    		"./test.yaml",
    	}))
    	select{}
    }
    

    I want to get real-time changed values via Event function , but it didn't do anything, what do I need to do to make it work?

  • 实现内存配置项备份特性

    实现内存配置项备份特性

    Is your feature request related to a problem? Please describe. no Describe the solution you'd like 1.框架提供一个备份选项,包括备份方式,备份周期等,当配置时,框架将自动进行备份 2.当开启备份功能时,框架初始化时,先load备份数据,然后在拉取其他的配置source 3.备份功能提供扩展功能,允许客户自行定义备份存放的地方(如OBS,本地,k8s的configmap等)

    Describe alternatives you've considered no

    Additional context no

    user story 作为框架的client,希望框架具有配置备份功能,这样当所以远端配置服务中心无法服务时,能够让服务在启动时先使用备份的配置项进行工作,等远端的配置中心恢复后再更新配置项

  • Ultilize Module Event Listener

    Ultilize Module Event Listener

    1. make CustomInit change the running status of archaius
    2. Do not dispatch events when dispatch module events
    3. correct the process of dispatch module, with a prefix tree
  • ssl不支持有bool类型的参数

    ssl不支持有bool类型的参数

    Describe the bug A clear and concise description of what the bug is.

    Version of go archaius v1.3.2 To Reproduce Steps to reproduce the behavior: ssl: rest.Provider.verifyPeer: false rest.Provider.cipherPlugin: go-lib-aes rest.Provider.cipherSuits: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 rest.Provider.protocol: TLSv1.2 rest.Provider.caFile: xxx rest.Provider.certFile: xxx rest.Provider.keyFile: xxx rest.Provider.certPwdFile: xxx

    存在bool类型的,就会出现读取不到这些配置情况,导致证书加载部分异常

    代码异常部分 unmarshal.go里面的populateMap函数

    		// maybe next map type,此时mapValueType 为string,而 setVal.Type() 为bool,导致函数退出,没有获取完需要的值
    		if mapValueType != setVal.Type() {
    			return rValue, nil
    
    		}
    

    Logs

  • Improvement expand value env

    Improvement expand value env

    优化yaml文件配置项中环境变量的替换,使得配置项可以进行配置项拼接,更符合shell语法,

    eg: chassis.yaml :

    addr1: ${IP||127.0.0.1}:${PORT||80}
    addr2: 0.0.0.0:${PORT||80}
    

    如果配置了环境变量PORT=443和IP=10.10.10.10,获得的结果:

    addr1: 10.10.10.10:443
    addr2: 0.0.0.0:443
    

    如果只配置了环境变量PORT=443,获得的结果:

    addr1: 127.0.0.1:443
    addr2: 0.0.0.0:443
    
  • event可以监控配置中心的namespace吗?

    event可以监控配置中心的namespace吗?

    https://github.com/go-chassis/go-archaius/blob/96fc4ada14924a9ce2f2ab0fc39c5e52e0cf0188/core/core.go#L68-L73

    https://github.com/go-chassis/go-archaius/blob/96fc4ada14924a9ce2f2ab0fc39c5e52e0cf0188/core/config-manager/configurationmanager.go#L396

    从这两处看event的key貌似就是配置信息的key。

    event有办法做到监控整个文件吗?比如配置中心中有个namespace叫test,能注册针对这个test的event吗,而不仅仅是针对其中的某个键值对。

  • fixed bug:Fix the problem that some types of yaml configuration cannot be converted, resulting in configuration loss

    fixed bug:Fix the problem that some types of yaml configuration cannot be converted, resulting in configuration loss

    修复yaml配置部分类型无法转换导致配置丢失的问题 比如 yaml配置value为bool类型 而接受类型为stirng 此时会将 bool转成stirng https://github.com/go-chassis/go-archaius/issues/122 https://github.com/go-chassis/go-chassis/issues/991

  • examples/event示例:配置刷新失败

    examples/event示例:配置刷新失败

    我想结合这个示例看一下Event部分的源码,但是示例跑的似乎有点问题:配置更改后,刷新失败。 运行环境:win10,IDE:Goland

    操作:将配置文件中age的值由13改为132 日志如下: 2019/04/15 14:30:50 13 2019/04/15 14:30:55 13 2019/04/15 14:31:00 13 2019/04/15 14:31:01 DEBUG: file event [E:\microservice\go-archaius-0.14.0\examples\event\event.yaml RENAME], operation is %!d(MISSING). reload it. 2019/04/15 14:31:01 WARN: [[E:\microservice\go-archaius-0.14.0\examples\event\event.yaml open E:\microservice\go-archaius-0.14.0\examples\event\event.yaml: The system cannot find the file specified.]] file does not exist so not able to watch further 2019/04/15 14:31:01 DEBUG: file event [E:\microservice\go-archaius-0.14.0\examples\event\event.yaml___jb_old___ WRITE], operation is %!d(MISSING). reload it. 2019/04/15 14:31:01 DEBUG: Event generated events [[]] 2019/04/15 14:31:01 DEBUG: file event [E:\microservice\go-archaius-0.14.0\examples\event\event.yaml___jb_old___ REMOVE], operation is %!d(MISSING). reload it. 2019/04/15 14:31:01 WARN: the file change mode: ["E:\microservice\go-archaius-0.14.0\examples\event\event.yaml___jb_old___": REMOVE], continue 2019/04/15 14:31:05 13 2019/04/15 14:31:10 13

  • github.com/go-chassis/go-archaius@v0.0.0-20180831094429-ab75db7118a6: invalid pseudo-version: does not match version-control timestamp (expected 20180927015433)

    github.com/go-chassis/[email protected]: invalid pseudo-version: does not match version-control timestamp (expected 20180927015433)

  • Cache config value to optimize Get config API

    Cache config value to optimize Get config API

    读配置的过程, 前后要经过好多次锁,而且会受 source 实现影响。改为直接缓存当前的配置值,读取时从当前缓存读。

    修改前:

    go test -bench=^Benchmark -benchtime=5s
    goos: darwin goarch: amd64 pkg: github.com/go-chassis/go-archaius/benchmark BenchmarkFileSource-8 5213296 1142 ns/op BenchmarkFileSourceParallelism-8 18202258 347 ns/op

    修改后:

    go test -bench=^Benchmark -benchtime=5s
    goos: darwin goarch: amd64 pkg: github.com/go-chassis/go-archaius/benchmark BenchmarkFileSource-8 68292421 87.3 ns/op BenchmarkFileSourceParallelism-8 100000000 54.8 ns/op

  • configmap can not be UnmarshalConfig

    configmap can not be UnmarshalConfig

    Describe the bug A clear and concise description of what the bug is. use archaius to parse configmap

            apiVersion: v1
            kind: ConfigMap
            metadata:
              name: roboartisan-robooperationmanagement-config
              labels:
                name: roboartisan-robooperationmanagement-config
            data:
                database.ip: 0.0.0.0
                database.port: 5432
                database.user_name: root
    

    the struct database

    type Database struct {
    	UserName string `json:"user_name"`
    	Password string `json:"password"`
    	IP       string `json:"ip"`
    	Port     int    `json:"port"`
    	DBName   string `json:"db_name"`
    	SslMode  string `json:"ssl_mode"`
    	MaxIdle  int    `json:"max_idle"`
    	MaxConn  int    `json:"max_conn"`
    }
    

    After archaius.UnmarshalConfig ,string type data can be parse, but int type data fail to parse
    Version of go chassis v1.5.4 To Reproduce Steps to reproduce the behavior:

    Logs #``

  • apollo 配置支持解析 yaml json 等

    apollo 配置支持解析 yaml json 等

    apollo config.yaml 命名空间配置示例

    a:
      b:
        c: 1
    

    apollo 支持 yaml json 等格式的命名空间, 目前的做法是直接作为一个配置, 如 config.yaml 命名空间会被读取成 key 为 config.yaml.content, value 为 yaml 内容的一个配置.

    希望可以实现解析, 按照 a.b.c 这样的方式直接读取到配置内容

  • 当获取不到配置的时候,指针类型应该赋予nil

    当获取不到配置的时候,指针类型应该赋予nil

    https://github.com/go-chassis/go-archaius/blob/4b83c04a7796963e1a83b70e0f699f5c7a44093a/source/unmarshal.go#L91

    在使用UnmarshalConfig的时候,如果yaml文件或者任意一个配置源获取不到配置,该项字段应该赋予零值,指针类型赋予nil。

Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

Dec 30, 2022
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Dec 24, 2019
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Jan 1, 2023
A distributed system for embedding-based retrieval
A distributed system for embedding-based retrieval

Overview Vearch is a scalable distributed system for efficient similarity search of deep learning vectors. Architecture Data Model space, documents, v

Dec 30, 2022
💡 A Distributed and High-Performance Monitoring System. The next generation of Open-Falcon
💡 A Distributed and High-Performance Monitoring System.  The next generation of Open-Falcon

夜莺简介 夜莺是一套分布式高可用的运维监控系统,最大的特点是混合云支持,既可以支持传统物理机虚拟机的场景,也可以支持K8S容器的场景。同时,夜莺也不只是监控,还有一部分CMDB的能力、自动化运维的能力,很多公司都基于夜莺开发自己公司的运维平台。开源的这部分功能模块也是商业版本的一部分,所以可靠性有保

Jan 5, 2023
Verifiable credential system on Cosmos with IBC for Distributed Identities
Verifiable credential system on Cosmos with IBC for Distributed Identities

CertX This is a project designed to demonstrate the use of IBC between different zones in the Cosmos ecosystem for privacy preserving credential manag

Mar 29, 2022
A distributed and coördination-free log management system
A distributed and coördination-free log management system

OK Log is archived I hoped to find the opportunity to continue developing OK Log after the spike of its creation. Unfortunately, despite effort, no su

Dec 26, 2022
A distributed MySQL binlog storage system built on Raft
A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

Dec 31, 2022
A distributed key-value storage system developed by Alibaba Group

Product Overview Tair is fast-access memory (MDB)/persistent (LDB) storage service. Using a high-performance and high-availability distributed cluster

Dec 31, 2022
Dkron - Distributed, fault tolerant job scheduling system https://dkron.io
Dkron - Distributed, fault tolerant job scheduling system https://dkron.io

Dkron - Distributed, fault tolerant job scheduling system for cloud native environments Website: http://dkron.io/ Dkron is a distributed cron service,

Dec 28, 2022
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

Jan 4, 2023
implementation of some distributed system techniques

Distributed Systems These applications were built with the objective of studding a distributed systems using the most recent technics. The main ideia

Feb 18, 2022
An distributed system developed by gin
An distributed system developed by gin

An distributed system developed by gin

Nov 22, 2021
The repository aims to share some useful about distributed system

The repository aims to share some useful about distributed system

Dec 14, 2021
Distributed-system - Practicing and learning the foundations of DS with Go

Distributed-System For practicing and learning the foundations of distributed sy

May 4, 2022
BlobStore is a highly reliable,highly available and ultra-large scale distributed storage system

BlobStore Overview Documents Build BlobStore Deploy BlobStore Manage BlobStore License Overview BlobStore is a highly reliable,highly available and ul

Oct 10, 2022
Go Micro is a framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Jan 8, 2023