A distributed MySQL binlog storage system built on Raft

Go Report Card Build Status

What is kingbus? 中文

Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the slaves in the same way as an intermediate MySQL master does. Kingbus has the following key features:

  • MySQL replication protocol compatibility, pull the binlog files from the master through gtid mode, and push the binlog file to slave through gtid mode in the same way.

  • Geo-Replication, kingbus uses Raft to support Geo-Replication. The binlog data written to the cluster is guaranteed to be consistent between multiple nodes, and the order of binlog event is exactly the same as that on the master.

  • High availability, your mysql binlog replication is always on and continuously available with kingbus.

Why need kingbus?

In a traditional MySQL replication setup a single master server is created and a set of slaves of MySQL servers are configured to pull the binlog files from the master, putting a lot of load on the master.

  • Introducing a layer between the master server and the slave servers can reduce the load on the master by only serving kingbus instead of all the slaves.

  • The slaves will only need to be aware of kingbus and not the real master server. Removing the requirement for the slaves to have knowledge of the master also simplifies the process of replacing a failed master within a replication environment.

  • Kingbus allows us to horizontally scale our slaves without fear of overloading the network interface of the master

  • Kingbus can also be used to avoide deep nested replication on remote sites, with kingbus you don't need e deeply nested replication.

  • The size of the binlog storage space on the Master can be reduced, and store the binlog in kingbus.

  • Support MYSQL database heterogeneous log based replication. Other heterogeneous replication components can be connected to the kingbus, such as canal.

For more usage scenarios of binlog server, please refer:

Quick start

Read the Quick Start

Documentation

1.Kingbus management API introduction

2.Start a kingbus cluster with docker-compose

License

Kingbus is under the Apache 2.0 license. See the LICENSES file for details.

Acknowledgments

  • Thanks etcd for providing raft library.

  • Thanks go-mysql for providing mysql replication protocol parser.

Owner
Fei Chen
Familiar with distributed system architecture,MySQL,Redis,Rocksdb,Kafka,Raft/Paxos etc
Fei Chen
https://github.com/flike/kingbus
Comments
  • [Question] Howto change kingbus setting when real master mysql is down

    [Question] Howto change kingbus setting when real master mysql is down

    Hi there,

    Sorry, I don't know where I should ask the question.

    I had 3 nodes kingbus and 10 slaves node. Kingbus was binlog server real master A. Master B was a slave master A. Now, we have some issues with master A and we want to change the real master for kingbus cluster. Should we do now? What is the script in the situation?

    +--------+                                                         +----------+
|master A| ------------replicated----------------------------------| master B |
+--------+                                                         +----------+
  ^
  |
  |

[ kingbus 1-3] <---------------------replicate [slave 1-10]

    Thank you!

  • 同步延迟不会被放大吗？

    同步延迟不会被放大吗？

    看了代码跟设计架构，作为一个中间件直接介入到数据库集群中，感觉存在这样几个问题：

    1. 引入运维复杂度跟风险点，维护数据库集群的时候需要再关注kingbus集群的状态。
    2. 同步复杂度增加，同样由于一个raft集群的引入，binlog的传输从之前的点对点变为了点到集群再到点，中间还夹杂着raft一致性保证带来的写放大的问题。即使是内网环境，随着kingbus节点和业务的增长，同步性能与数据安全性的平衡怎么保证？写入量大的场景备份延迟如何保证不能被放大？
    3. kingbus集群leader挂掉，等同于所有的从库需要做一次主从切换，并没有减少切换成本。

    感觉这个东西做个binlog server来备份、分发数据什么的还可以，至少比canel少了一堆组件要部署，但是作为中间件连接master和slave，感觉稍微有些鸡肋啊。

  • 是否支持增强半同步复制

    是否支持增强半同步复制

    在异步复制架构中，master宕机时，如果部分binlog event未及时发送，会导致事务丢失。同样的，在使用mysqlbinlog工具做binlog server时，也会存在这种问题，并且由于操作系统机制，可能会有更多的情况产生。 请问，该工具能否支持半同步复制，或者是否有额外的措施用以弥补上述缺陷，谢谢！

  • 从库无法获取serverid 导致无法同步

    从库无法获取serverid 导致无法同步

    slave 报错 Last_IO_Errno: 1105 Last_IO_Error: The slave I/O thread stops because a fatal error is encountered when it try to get the value of SERVER_ID variable from master. Error: mysql:sql not support

    kingbus报错

    2019-07-08T15:50:00.397+0800 info mysql/command.go:119 handleQuery sql:SELECT UNIX_TIMESTAMP() 2019-07-08T15:50:00.398+0800 info mysql/command.go:119 handleQuery sql:SHOW VARIABLES LIKE 'SERVER_ID' 2019-07-08T15:50:00.399+0800 debug mysql/command.go:283 show query showVariable: 2019-07-08T15:50:00.399+0800 error mysql/conn.go:151 dispatch error,err:mysql:sql not support,data:[3 83 72 79 87 32 86 65 82 73 65 66 76 69 83 32 76 73 75 69 32 39 83 69 82 86 69 82 95 73 68 39] 2019-07-08T15:50:00.399+0800 debug mysql/conn.go:135 close client connection

    如何解决？

  • 关于mysql配置的疑问

    关于mysql配置的疑问

    通过代码发现：syncer获取mysql master的连接配置，是通过raft协议传递的，而不是通过配置的方式，通过代码暴露出的http接口，也没有找到能通过http设置mysql的连接配置的地方，请问该连接配置，怎么通知集群的？

    BinlogServer的启动时通过http接口，用户主动开启，然后等待mysql slave的tcp 连接。

Related tags
Database mysql golang distributed-systems storage raft mysql-binlog binlog
The MySQL Cluster Autopilot Management with GTID and Raft
 The MySQL Cluster Autopilot Management with GTID and Raft

Xenon Overview Xenon is a MySQL HA and Replication Management tool using Raft protocol. Xenon has many cool features, such as: Fast Failover with no l

Jan 3, 2023
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

Jan 9, 2023
Beerus-DB: a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic operations

Beerus-DB · Beerus-DB is a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic

Oct 29, 2022
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice. A simple in-memory database and table implementation are provided, and you can query any data source you want by implementing a few interfaces.

Dec 27, 2022
TiDB is an open source distributed HTAP database compatible with the MySQL protocol
TiDB is an open source distributed HTAP database compatible with the MySQL protocol

Slack Channel Twitter: @PingCAP Reddit Mailing list: lists.tidb.io For support, please contact PingCAP What is TiDB? TiDB ("Ti" stands for Titanium) i

Jan 9, 2023
Secure storage for personal records built to comply with GDPR
 Secure storage for personal records built to comply with GDPR

Databunker Databunker is a Personally Identifiable Information (PII) Data Storage Service built to Comply with GDPR and CCPA Privacy Requirements. Pro

Jan 8, 2023
Multitiered file storage API built on Filecoin and IPFS
 Multitiered file storage API built on Filecoin and IPFS

Powergate Powergate is a multitiered file storage API built on Filecoin and IPFS, and an index builder for Filecoin data. It's designed to be modular

Dec 20, 2022
Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding.

Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding.

Jan 4, 2023
The lightweight, distributed relational database built on SQLite.
 The lightweight, distributed relational database built on SQLite.

rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it grace

Jan 5, 2023
RadonDB is an open source, cloud-native MySQL database for building global, scalable cloud services

OverView RadonDB is an open source, Cloud-native MySQL database for unlimited scalability and performance. What is RadonDB? RadonDB is a cloud-native

Dec 31, 2022
Run MySQL Database on Docker

Run MySQL Database on Docker cd <path>/resources/docker sudo docker-compose up (sudo for linux) This will start a container MySQL Database running on

Jan 1, 2022
Virsas-mod-db - Quick way to init mysql, postgres and redis connection from multiple services without duplicating the code

Quick way to init mysql, postgres and redis connection from multiple services without duplicating the code.

Jan 23, 2022
moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library. moss stands for "memory-oriented s

Dec 18, 2022
A GPU-powered real-time analytics storage and query engine.
A GPU-powered real-time analytics storage and query engine.

AresDB AresDB is a GPU-powered real-time analytics storage and query engine. It features low query latency, high data freshness and highly efficient i

Jan 7, 2023
Key-Value Storage written in Go.

kvs kvs is an in-memory key-value storage written in Go. It has 2 different usage. It can be used as a package by importing it to your code or as a se

Jun 15, 2022
Golang in-memory database built on immutable radix trees

go-memdb Provides the memdb package that implements a simple in-memory database built on immutable radix trees. The database provides Atomicity, Consi

Jan 7, 2023
Eventually consistent distributed in-memory cache Go library

bcache A Go Library to create distributed in-memory cache inside your app. Features LRU cache with configurable maximum keys Eventual Consistency sync

Dec 2, 2022
CockroachDB - the open source, cloud-native distributed SQL database.
 CockroachDB - the open source, cloud-native distributed SQL database.

CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. What is CockroachDB? Docs Quickstart C

Jan 2, 2023
A distributed key-value store. On Disk. Able to grow or shrink without service interruption.

Vasto A distributed high-performance key-value store. On Disk. Eventual consistent. HA. Able to grow or shrink without service interruption. Vasto sca

Jan 6, 2023