A simple distributed kv system from scratch

SimpleKV

A simple distributed key-value storage system based on bitcask from scratch.

Target

Here are some basic requirements:

  • LRU Cache.
  • An index system based on either hashmap / skiplist.
  • Support string data structure. Implement GET/SET/EXPIRE/DEL method.
  • Appendonly log as disk storage with Bitcask theory.
  • Master-slave replication.
  • Using zookeeper/etcd as service registration, leader election, etc.
  • Client-side router cache and load balance.
  • Using GRPC and singleflight mechanism to optimize traffic.
  • Using Docker to put all stuff together.

Advanced requirements:

  • Multi storage unit using hashslot / concurrency hash.
  • Achieve elastic expansion.
  • Frontend dashboard.
  • Support list / hash / set / zset data structure.

Special Thanks

Some projects I've studied which inspired this project.

  • rosedb: a kv system based on bitcask.
  • minidb: a minimum implementation of bitcask, written by the same author for tutorial purpose.
  • groupcache: distributed cache system using concurrency hash + singleflight.
  • secret project A: a distributed kv system based on storage unit consists of rocksdb + LRU cache, using hashslot to achieve distributed storage and elastic expansion.
  • bitcask paper: https://riak.com/assets/bitcask-intro.pdf
Owner
Galaxyzeta
Undergraduate student who loves cool stuff. Indie developer, novice music composer and a Nintendo fan for 15 years : )
Galaxyzeta
https://github.com/Galaxyzeta/simplekv
Similar Resources

The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Jul 26, 2022

Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm

Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the R

Dec 7, 2022

Simple, ordered, key-value persistence library for the Go Language

gkvlite gkvlite is a simple, ordered, ACID, key-value persistence library for Go. Overview gkvlite is a library that provides a simple key-value persi

Dec 21, 2022

moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library.

Dec 18, 2022

yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.

yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.

yakv (yak-v. (originally intended to be "yet-another-key-value store")) is a simple, in-memory, concurrency-safe key-value store for hobbyists. yakv provides persistence by appending transactions to a transaction log and restoring data from the transaction log on startup.

Feb 24, 2022

A simple batching library in Golang

Go Batch A simple batching library in Golang. Guid Installation go get github.com/RashadAnsari/go-batch Example package main import ( "log" "math/r

Aug 12, 2022

ShockV is a simple key-value store with RESTful API

ShockV is a simple key-value store with RESTful API

ShockV is a simple key-value store based on badgerDB with RESTful API. It's best suited for experimental project which you need a lightweight data store.

Sep 26, 2021

simple kv store engine inspired by bitcask

tiny-bitcask simple kv store engine inspired by bitcask Inspired By BitCask BitCask System Design DB启动时，系统内存储着merged-data-file 与 hint-file 系统通过读取hint-

Nov 14, 2021

Simple in memory key-value store.

Simple in memory key-value store. Development This project is written in Go. Make sure you have Go installed (download). Version 1.17 or higher is req

Nov 6, 2021
Related tags
Key-Value Store simplekv
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.

Consul Website: https://www.consul.io Tutorials: HashiCorp Learn Forum: Discuss Consul is a distributed, highly available, and data center aware solut

Jan 2, 2023
Implementation of distributed key-value system based on TiKV

Distributed_key-value_system A naive implementation of distributed key-value system based on TiKV Features Features of this work are listed below: Dis

Mar 7, 2022
Simple Distributed key-value database (in-memory/disk) written with Golang.

Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get

Jan 18, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Jan 4, 2023
Distributed, fault-tolerant key-value storage written in go.
 Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Jan 3, 2023
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
 GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jan 6, 2023
Distributed key-value store
 Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Nov 15, 2021
rgkv is a distributed kv storage service using raft consensus algorithm.
 rgkv is a distributed kv storage service using raft consensus algorithm.

rgkv rgkv is a distributed kv storage service using raft consensus algorithm. Get/put/append operation High Availability Sharding Linearizability Tabl

Jan 15, 2022
Distributed disk storage database based on Raft and Redis protocol.
 Distributed disk storage database based on Raft and Redis protocol.

IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and

Dec 27, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Jan 4, 2022