rgkv is a distributed kv storage service using raft consensus algorithm.

rgkv

rgkv is a distributed kv storage service using raft consensus algorithm.

  • Get/put/append operation
  • High Availability
  • Sharding
  • Linearizability

Table of Content

Test Case

Try to use go test to run test case

  • raft test (src/raft/test_test.go)
    cd src/raft/
    go test -run 
         
    
         
  • shardctrler test (src/shardctrler/test_test.go)
    cd src/shardctrler/
    go test -run 
         
    
         
  • shardkv test (src/shardkv/test_test.go)
    cd src/shardkv/
    go test -run 
         
    
         

Architecture

rgkv architecture

  • shard controller

    The shardctrler manages a sequence of numbered configurations. Each configuration describes a set of replica groups and an assignment of shards to replica groups. Shard controller use raft to prevent single-point failure.

    Support API:

    • Query: Retrieve config with a config number. if config number is -1, get the latest config
    • Join: Add a new replica group
    • Leave: Leave a list of replica groups
    • Move: assign a shard to a certain replica group
  • shardkv server

    Each shardkv server operates as part of a raft replica group. Each replica group serves Get/Put/Append API for some of the shards.

    Support API:

    • Get
    • Put
    • Apppend
  • shards movement

    rgkv uses a way similar to two phases commit to implement shards movement logic. It ensures correct movement logic, garbage collection for stale shards, and uninterrupted service during movement.

    • phase 1: shard controller sends prepare request to kv servers to get shards they need and install shards by sending it to raft log
    • phase 2: shard controller sends commit request to kv servers to garbage collect stale shards.

    *each shard carry a config number to prevent tasks that have already been executed from being repeated

Raft

raft is a distributed consensus algorithm, which is easy to understand. The raft library in rgkv supported:

  • Leader election
  • Log
  • Persistence
  • Snapshot

design diagram

raft diagram

Linearizability

Linearizability ensured by:

  • Read/get/append operations order is determined by raft log
  • Duplicate client request detection mechanism (but each client can only send req sequentially, wait for optimize)
Similar Resources

A key-value storage transaction interpretator.

kv-txn-interpreter A key-value storage transaction interpreter, which provides an etcd-like transaction interface to help you build a transaction over

Feb 22, 2022

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.

Dec 28, 2022

Keyval - A simple key-value storage library written in Go

keyval keyval is a simple key-value storage library written in Go and its back c

Sep 16, 2022

GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jan 6, 2023

Distributed key-value store

Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Nov 15, 2021

A simple distributed kv system from scratch

SimpleKV A simple distributed key-value storage system based on bitcask from scratch. Target Here are some basic requirements: LRU Cache. An index sys

Apr 21, 2022

Simple Distributed key-value database (in-memory/disk) written with Golang.

Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get

Jan 18, 2022

Implementation of distributed key-value system based on TiKV

Distributed_key-value_system A naive implementation of distributed key-value system based on TiKV Features Features of this work are listed below: Dis

Mar 7, 2022

Golang-key-value-store - Key Value Store API Service with Go DDD Architecture

This document specifies the tools used in the Key-Value store and reorganizes how to use them. In this created service, In-Memory Key-Value Service was created and how to use the API is specified in the HTML file in the folder named "doc"

Jul 31, 2022
Related tags
The TinyKV course builds a key-value storage system with the Raft consensus algorithm
The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Jul 26, 2022
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the R

Dec 7, 2022
Distributed disk storage database based on Raft and Redis protocol.
Distributed disk storage database based on Raft and Redis protocol.

IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and

Dec 27, 2022
Distributed, fault-tolerant key-value storage written in go.
Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Jan 3, 2023
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

Dec 28, 2022
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.

Consul Website: https://www.consul.io Tutorials: HashiCorp Learn Forum: Discuss Consul is a distributed, highly available, and data center aware solut

Jan 2, 2023
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Jan 4, 2023
moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library.

Dec 18, 2022
A key-value db api with multiple storage engines and key generation
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

Apr 5, 2022
Fault tolerant, sharded key value storage written in GoLang
Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Nov 1, 2022