rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.

rosedb_ico.png

 Go Report Card GitHub top language GitHub stars codecov CodeFactor Go Reference Mentioned in Awesome Go

English | 简体中文

rosedb is not production-ready, we are doing more test now.

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.

rosedb supports a variety of data structures such as string, list, hash, set and sorted set.

rosedb is written in pure Go, easy to understand and use.

Our vision is to build an efficient k-v storage engine for Go-based applications. Feel free to give us any advices, and also please give us a star ,that helps a lot, thanks!

Stargazers over time

Features

  • Supports all common data structures : string, list, hash, set, zset.
  • Easy to embed (import "github.com/roseduan/rosedb").
  • Low latency and high throughput (see Benchmark).
  • Has built-in parallel execution of data modification on many provided data structures.
  • Comes with rosedb-cli for command line access, that is compatible with redis-cli.
  • Supports TTL-based key eviction.
  • Supports prefix scan and range scan for string keys.
  • Support simple transaction, ACID features.
  • Merge operation can be stopped manually.

Usage

Cli example

Navigate to rosedb/cmd/server and run main.go

Xnip2021-04-14_14-33-11.png

Open a new terminal, navigate to rosedb/cmd/cli, and run main.go

Xnip2021-04-14_14-35-50.png

Or you can just use redis-cli or any other redis client:

2021-05-14 上午11.19.24.png

Embedding example

Import rosedb in the application:

import "github.com/roseduan/rosedb"

Open a connection to the database:

package main

import (
	"github.com/roseduan/rosedb"
	"log"
)

func main() {
	config := rosedb.DefaultConfig()
	db, err := rosedb.Open(config)
	
	if err != nil {
		log.Fatal(err)
	}
	
  // don`t forget to close!
	defer db.Close()
	
	//...
}

Deploy and run on Docker

set hello rosedb OK 127.0.0.1:5200>get hello rosedb ">
docker build -t="rosedb:v1.2.9" .
docker run --name=rosedb -itd -p 5200:5200 rosedb:v1.2.9
docker exec -it rosedb sh

$ rosedb-cli
127.0.0.1:5200>set hello rosedb
OK
127.0.0.1:5200>get hello
rosedb

Command

String

  • Set
  • SetNx
  • Get
  • GetSet
  • Append
  • StrLen
  • StrExists
  • StrRem
  • PrefixScan
  • RangeScan
  • Expire
  • Persist
  • TTL

List

  • LPush
  • RPush
  • LPop
  • RPop
  • LIndex
  • LRem
  • LInsert
  • LSet
  • LTrim
  • LRange
  • LLen
  • LKeyExists
  • LValExists

Hash

  • HSet
  • HSetNx
  • HGet
  • HGetAll
  • HDel
  • HExists
  • HLen
  • HKeys
  • HVals

Set

  • SAdd
  • SPop
  • SIsMember
  • SRandMember
  • SRem
  • SMove
  • SCard
  • SMembers
  • SUnion
  • SDiff

Zset

  • ZAdd
  • ZScore
  • ZCard
  • ZRank
  • ZRevRank
  • ZIncrBy
  • ZRange
  • ZRangeWithScores
  • ZRevRange
  • ZRevRangeWithScores
  • ZRem
  • ZGetByRank
  • ZRevGetByRank
  • ZScoreRange
  • ZRevScoreRange

TODO

see here.

Benchmark

Environment

  • Go version:1.14.4
  • OS: macOS Catalina 10.15.7
  • CPU: 2.6GHz 6-Core Intel Core i7
  • Memory: 16 GB 2667 MHz DDR4
  • Code: rosedb-bench
  • Other databases for comparison:
    • Badger
    • GoLevelDB
    • Pudge

Result

1,000,000 iterations

go test -bench=. -benchtime=1000000x
badger 2021/05/16 21:59:53 INFO: All 0 tables opened in 0s
badger 2021/05/16 21:59:53 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 21:59:53 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    1000000             11518 ns/op            2110 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    1000000              3547 ns/op            1172 B/op         20 allocs/op
BenchmarkPutValue_GoLevelDB-12                   1000000              4659 ns/op             352 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   1000000              2838 ns/op             814 B/op         13 allocs/op
BenchmarkPutValue_Pudge-12                       1000000              8512 ns/op             791 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       1000000              1253 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            1000000              4371 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            1000000               481 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           1000000              4255 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           1000000              2986 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    46.388s

2,500,000 iterations

go test -bench=. -benchtime=2500000x
badger 2021/05/16 22:06:08 INFO: All 0 tables opened in 0s
badger 2021/05/16 22:06:08 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 22:06:08 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    2500000             11660 ns/op            2150 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    2500000              4180 ns/op            1222 B/op         21 allocs/op
BenchmarkPutValue_GoLevelDB-12                   2500000              4637 ns/op             336 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   2500000              2942 ns/op             817 B/op         14 allocs/op
BenchmarkPutValue_Pudge-12                       2500000              9238 ns/op             763 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       2500000              1275 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            2500000              4474 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            2500000               525 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           2500000              4294 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           2500000              3038 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    119.529s

Conclusion

Badger

Stable read and write performance. Write: 11000+ ns/op. Read: 4000+ ns/op.

GoLevelDB

Write performance is almost 2.5x that of Badger. Read is almost 3000 ns/op, a little faster than that of Badger.

Pudge

Write performance is between that of GoLevelDB and Badger, almost 8500 ns/op. Read is very fast and stable, almost twice as fast as that of GoLevelDB.

RoseDB

Write performance is stable, almost the same as that of GoLevelDB, 2.5x that of Badger.

In KeyValueRamMode, since the values are all in memory, it is the fastest of all.

In KeyOnlyRamMode, it is almost the same as GoLevelDB.

Contributing

If you are intrested in contributing to rosedb, please see here: CONTRIBUTING

Contact me

If you have any questions, you can contact me by email: [email protected]

License

rosedb is licensed under the term of the MIT License

Owner
roseduan
微信公众号:roseduan写字的地方,回复 Go,你懂的。
roseduan
Comments
  • Add Decr, DecrBy, Incr, IncrBy commands

    Add Decr, DecrBy, Incr, IncrBy commands

  • 修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改更换ActiveFile时机(Modify the timing of replacing ActivaFile)

    当前实现(Now Impl)

    • 判断当前entry是否能够写入ActiveFile(Determine whether the current entry can be written to ActiveFile)
    • 如果不能则更换ActiveFile(If not, change ActiveFile)
    • 将Entry写入到新的ActiveFile中(Write Entry to new ActiveFile)

    修改(Modify)

    • 对于当前entry, 如果ActiveFile空间不足,将超出BlockSize限制将entry写入到当前ActiveFile中(For the current entry, if the ActiveFile space is insufficient, the entry will be written to the current ActiveFile beyond the BlockSize limit)
    • 采用类似懒加载方式。每次写时先检查上次写完之后ActiveFile文件是否已满, 如果满那么写之前更换ActiveFile。(Just like lazy loading, each time you write, first check whether the ActiveFile file is full after the last write is completed, if it is full, then replace the ActiveFile before writing)

    好处(Advantage)

    参照MySQL binlog处理方式,能够最大化利用空间(Imitate MySQL binlog processing method, can maximize the use of space)

  • Add LRange commad

    Add LRange commad

    Related issue: https://github.com/flower-corp/rosedb/issues/149

    • refactor listSequence to apply it in LRange function
    • add listSequence test cases
    • Add LRange function
  • Add SDiff command

    Add SDiff command

    • SDiff command implemented. SMembers() logic part exported as private helper method in order to use it another method(e.g., inside of the SDiff() method).
    • Tests added for SDiff().

    Signed-off-by: Gökhan Özeloğlu [email protected]

  • build problems

    build problems

    Hello,

    Using: go version go1.16 windows/amd64

    I just tried to build RoseDB, but have errors:

    C:\goprojects\src\RoseDB\test>go build cmd\server\main.go cmd\grpc_server.go:6:2: no required module provides package github.com/roseduan/rosedb/cmd/proto; to add it: go get github.com/roseduan/rosedb/cmd/proto

    C:\goprojects\src\RoseDB\test>go get github.com/roseduan/rosedb/cmd/proto go get github.com/roseduan/rosedb/cmd/proto: no matching versions for query "upgrade"

  • lpop or rpop命令每删除一个数据,发送两次discard数据到discard文件

    lpop or rpop命令每删除一个数据,发送两次discard数据到discard文件

    Describe the bug https://github.com/flower-corp/rosedb/blob/main/list.go#L381

    	if err = db.saveListMeta(idxTree, key, headSeq, tailSeq); err != nil {
    		return nil, err
    	}
    	// send discard
    	db.sendDiscard(oldVal, updated, List)                                                 // 第一次发送到discard
    	_, entrySize := logfile.EncodeEntry(ent)
    	node := &indexNode{fid: pos.fid, entrySize: entrySize}
    	select {
    	case db.discards[List].valChan <- node:                                           // 第二次发送到discard
    	default:
    		logger.Warn("send to discard chan fail")
    	}
    

    我认为删除一条entry数据,仅需要发送一次discard文件,因此我想仅保存db.sendDiscard(oldVal, updated, List),此处是否为一个bug? To Reproduce Steps to reproduce the behavior: NA

    Expected behavior NA

    Additional context NA

  • why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    For structs in the project, such as "listIndex or hashIndex", mu is defined as the pointer type *sync. RWMutex, in the project mu is defined as is there anything wrong with sync.RWMutex(e.g. sync. RWMutex copy problem)?

  • Sharding for P2P model?

    Sharding for P2P model?

    Hello,

    I have just come across RoseDB and think that it might be what I need for a project that I am working on.

    One thing that I am needing is to develop for a P2P solution for the database with high number of nodes and am wondering about what it might take to have simple sharding added to RoseDB so that it can do high node clustering and spread out the data across all nodes as needed?

  • 退出GC程序的signal信号问题

    退出GC程序的signal信号问题

    Describe the bug https://github.com/flower-corp/rosedb/blob/main/db.go#L490

    signal.Notify(quitSig, os.Interrupt, os.Kill, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
    

    问题:监听的信号os.Interrupt syscall.SIGINT存在重复

    To Reproduce NA

    Expected behavior NA

    Additional context NA

  • Why the func is legal

    Why the func is legal

    https://github.com/flower-corp/rosedb/blob/4a3aa6540cd8c043886aee5f9b425245860ad335/util/hash.go#L7 I have no idea why this func define is legal? there is no alias for it, how it work?

  • Add MSetNX command

    Add MSetNX command

    Equivalent Redis command: MSetNX

    • Helper function added to check duplicate keys.
    • Tests added.
    • Example usage added.

    Signed-off-by: Gökhan Özeloğlu [email protected]

  • Failed under ARM/x86 architecture

    Failed under ARM/x86 architecture

    Problem with alignment on ARM (32bit) systems

    runtime/internal/atomic.panicUnaligned()
    	.../runtime/internal/atomic/unaligned.go:8 +0x24
    runtime/internal/atomic.Store64(0x205416c, 0x0)
    	.../runtime/internal/atomic/atomic_arm.s:291 +0x14
    github.com/flower-corp/rosedb.(*RoseDB).loadIndexFromLogFiles.func1(0x0, 0x20ac630)
    	.../go/pkg/mod/github.com/flower-corp/[email protected]/index.go:208 +0x268
    created by github.com/flower-corp/rosedb.(*RoseDB).loadIndexFromLogFiles
    	.../go/pkg/mod/github.com/flower-corp/[email protected]/index.go:216 +0x88
    
  • feature: feat(feature): Add incrByFloat ()

    feature: feat(feature): Add incrByFloat ()

    I've been around roseDB for a long time, but this is the first time I've contributed. I've provided an incrByFloat() function for self-increment ids, and I'll try to continue contributing to roseDB and LotusDB, thanks

  • Add More Benchmark Test

    Add More Benchmark Test

    The performance of rosedb is not so clear. We don't know how it perform as a kv database, just add a benchmark like other kv db can help a lot to constract other kv database.

  • add GetAllKeys

    add GetAllKeys

    A new feature: list all stored keys conatain all datastructure types.

    Here is my idea:

    • add GetKeys command in different data types respectively(GetStrsKeys, GetListKeys, GetHashKeys...)
    • add GetAllKeys command in db.go, retrieve all data types GetKeys concurrently

    The GetAllKeys method like this: func (db *RoseDB) GetAllKeys(dataTypes... DataType) (map[DataType][][]byte, error)

simple kv store engine inspired by bitcask

tiny-bitcask simple kv store engine inspired by bitcask Inspired By BitCask BitCask System Design DB启动时,系统内存储着merged-data-file 与 hint-file 系统通过读取hint-

Nov 14, 2021
A key-value db api with multiple storage engines and key generation
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

Apr 5, 2022
Golang-key-value-store - Key Value Store API Service with Go DDD Architecture

This document specifies the tools used in the Key-Value store and reorganizes how to use them. In this created service, In-Memory Key-Value Service was created and how to use the API is specified in the HTML file in the folder named "doc"

Jul 31, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Jan 4, 2023
An embedded key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

Dec 30, 2022
Pogreb is an embedded key-value store for read-heavy workloads written in Go.
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Dec 29, 2022
CrankDB is an ultra fast and very lightweight Key Value based Document Store.

CrankDB is a ultra fast, extreme lightweight Key Value based Document Store.

Apr 12, 2022
Distributed, fault-tolerant key-value storage written in go.
Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Jan 3, 2023
Fault tolerant, sharded key value storage written in GoLang
Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Nov 1, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Jan 4, 2022
A key-value storage transaction interpretator.

kv-txn-interpreter A key-value storage transaction interpreter, which provides an etcd-like transaction interface to help you build a transaction over

Feb 22, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm
The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Jul 26, 2022
Keyval - A simple key-value storage library written in Go

keyval keyval is a simple key-value storage library written in Go and its back c

Sep 16, 2022
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the R

Dec 7, 2022
NutsDB a simple, fast, embeddable and persistent key/value store written in pure Go.
NutsDB a simple, fast, embeddable and persistent key/value store written in pure Go.

A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

Jan 9, 2023
Fast key-value DB in Go.
Fast key-value DB in Go.

BadgerDB BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast,

Jan 1, 2023
moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library.

Dec 18, 2022
HA LDAP based key/value solution for projects configuration storing with multi master replication support

Recon is the simple solution for storing configs of you application. There are no specified instruments, no specified data protocols. For the full power of Recon you only need curl.

Jun 15, 2022
Implementation of distributed key-value system based on TiKV

Distributed_key-value_system A naive implementation of distributed key-value system based on TiKV Features Features of this work are listed below: Dis

Mar 7, 2022