Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course

The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

After completing this course, you will have the knowledge to implement a horizontally scalable, highly available, key-value storage service with distributed transaction support. Also, you will have a better understanding of TiKV architecture and implementation.

Course Architecture

The whole project is a skeleton code for a key-value server and a scheduler server at the beginning - you need to finish the core logic step by step:

  • Standalone KV
    • Implement a standalone storage engine.
    • Implement raw key-value service handlers.
  • Raft KV
    • Implement the basic Raft algorithm.
    • Build a fault-tolerant KV server on top of Raft.
    • Add the support of Raft log garbage collection and snapshot.
  • Multi-raft KV
    • Implement membership change and leadership change to Raft algorithm.
    • Implement conf change and region split on Raft store.
    • Implement a basic scheduler.
  • Transaction
    • Implement the multi-version concurrency control layer.
    • Implement handlers of KvGet, KvPrewrite, and KvCommit requests.
    • Implement handlers of KvScan, KvCheckTxnStatus, KvBatchRollback, and KvResolveLock requests.

Code Structure

overview

Similar to the architecture of TiDB + TiKV + PD that separates the storage and computation, TinyKV only focuses on the storage layer of a distributed database system. If you are also interested in the SQL layer, please see TinySQL. Besides that, there is a component called TinyScheduler acting as a center control of the whole TinyKV cluster, which collects information from the heartbeats of TinyKV. After that, the TinyScheduler can generate scheduling tasks and distribute the tasks to the TinyKV instances. All of instances are communicated via RPC.

The whole project is organized into the following directories:

  • kv contains the implementation of the key-value store.
  • raft contains the implementation of the Raft consensus algorithm.
  • scheduler contains the implementation of the TinyScheduler, which is responsible for managing TinyKV nodes and generating timestamps.
  • proto contains the implementation of all communication between nodes and processes uses Protocol Buffers over gRPC. This package contains the protocol definitions used by TinyKV, and the generated Go code that you can use.
  • log contains utility to output log based on level.

Reading List

We provide a reading list for the knowledge of distributed storage system. Though not all of them are highly related with this course, they can help you construct the knowledge system in this field.

Also, you're encouraged to read the overview of TiKV's and PD's design to get a general impression on what you will build:

Build TinyKV from Source

Prerequisites

  • git: The source code of TinyKV is hosted on GitHub as a git repository. To work with git repository, please install git.
  • go: TinyKV is a Go project. To build TinyKV from source, please install go with version greater or equal to 1.13.

Clone

Clone the source code to your development machine.

git clone https://github.com/tidb-incubator/tinykv.git

Build

Build TiDB from the source code.

cd tinykv
make

It builds the binary of tinykv-server and tinyscheduler-server to bin dir.

Run TinyKV with TinySQL

  1. Get tinysql-server follow its document.
  2. Put the binary of tinyscheduler-server, tinykv-server and tinysql-server into a single dir.
  3. Under the binary dir, run the following commands:
mkdir -p data
./tinyscheduler-server
./tinykv-server -path=data
./tinysql-server --store=tikv --path="127.0.0.1:2379"

Now you can connect to the database with an official MySQL client:

mysql -u root -h 127.0.0.1 -P 4000

Contributing

Any feedback and contribution is greatly appreciated. Please see issues if you want to join in the development.

Similar Resources

Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

Dec 28, 2022

Implementation of distributed key-value system based on TiKV

Distributed_key-value_system A naive implementation of distributed key-value system based on TiKV Features Features of this work are listed below: Dis

Mar 7, 2022

An embedded key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

Dec 30, 2022

A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Jan 1, 2023

Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Jan 4, 2023

Simple, ordered, key-value persistence library for the Go Language

gkvlite gkvlite is a simple, ordered, ACID, key-value persistence library for Go. Overview gkvlite is a library that provides a simple key-value persi

Dec 21, 2022

LevelDB key/value database in Go.

This is an implementation of the LevelDB key/value database in the Go programming language. Installation go get github.com/syndtr/goleveldb/leveldb R

Jan 9, 2023

a persistent real-time key-value store, with the same redis protocol with powerful features

a persistent real-time key-value store, with the same redis protocol with powerful features

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Dec 25, 2022

GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jan 6, 2023
The TinyKV course builds a key-value storage system with the Raft consensus algorithm
The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Jul 26, 2022
rgkv is a distributed kv storage service using raft consensus algorithm.
rgkv is a distributed kv storage service using raft consensus algorithm.

rgkv rgkv is a distributed kv storage service using raft consensus algorithm. Get/put/append operation High Availability Sharding Linearizability Tabl

Jan 15, 2022
Golang-key-value-store - Key Value Store API Service with Go DDD Architecture

This document specifies the tools used in the Key-Value store and reorganizes how to use them. In this created service, In-Memory Key-Value Service was created and how to use the API is specified in the HTML file in the folder named "doc"

Jul 31, 2022
A key-value db api with multiple storage engines and key generation
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

Apr 5, 2022
Distributed, fault-tolerant key-value storage written in go.
Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Jan 3, 2023
Fault tolerant, sharded key value storage written in GoLang
Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Nov 1, 2022
A key-value storage transaction interpretator.

kv-txn-interpreter A key-value storage transaction interpreter, which provides an etcd-like transaction interface to help you build a transaction over

Feb 22, 2022
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.

Dec 28, 2022
Keyval - A simple key-value storage library written in Go

keyval keyval is a simple key-value storage library written in Go and its back c

Sep 16, 2022
Distributed disk storage database based on Raft and Redis protocol.
Distributed disk storage database based on Raft and Redis protocol.

IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and

Dec 27, 2022