rosedb is an embedded and fast k-v database based on LSM + WAL

rosedb_ico.png

 Go Report Card GitHub top language GitHub stars codecov CodeFactor Go Reference Mentioned in Awesome Go

English | 简体中文

rosedb is an embedded and fast k-v database based on LSM + WAL, so it has pretty good write performance and high throughput. It also supports many kinds of data structures such as string, list, hash, set, zset,and the API name style is similar to Redis.

rosedb is in pure Go, simple and easy to understand for using or learning.

Feature

  • Support rich data structures : string, list, hash, set, zset.
  • Easy to embedded (import "github.com/roseduan/rosedb").
  • Low latency and high throughput.
  • Operations of various data types can be parallel.
  • Has builtin rosedb-cli for command line, also support redis-cli.
  • Support expiration and TTL.

Usage

Cli example

Change the directory to rosedb/cmd/server.

Run the main.go

Xnip2021-04-14_14-33-11.png

Open a new shell, and change the directory to rosedb/cmd/cli, and run the main.go

Xnip2021-04-14_14-35-50.png

Or you can just use redis-cli or any other redis client:

2021-05-14 上午11.19.24.png

Embedded example

Import rosedb in the application:

import "github.com/roseduan/rosedb"

And open a database:

package main

import (
	"github.com/roseduan/rosedb"
	"log"
)

func main() {
	config := rosedb.DefaultConfig()
	db, err := rosedb.Open(config)
	
	if err != nil {
		log.Fatal(err)
	}
	
  // don`t forget to close!
	defer db.Close()
	
	//...
}

Command

String

  • Set
  • SetNx
  • Get
  • GetSet
  • Append
  • StrLen
  • StrExists
  • StrRem
  • PrefixScan
  • RangeScan
  • Expire
  • Persist
  • TTL

List

  • LPush
  • RPush
  • LPop
  • RPop
  • LIndex
  • LRem
  • LInsert
  • LSet
  • LTrim
  • LRange
  • LLen

Hash

  • HSet
  • HSetNx
  • HGet
  • HGetAll
  • HDel
  • HExists
  • HLen
  • HKeys
  • HValues

Set

  • SAdd
  • SPop
  • SIsMember
  • SRandMember
  • SRem
  • SMove
  • SCard
  • SMembers
  • SUnion
  • SDiff

Zset

  • ZAdd
  • ZScore
  • ZCard
  • ZRank
  • ZRevRank
  • ZIncrBy
  • ZRange
  • ZRevRange
  • ZRem
  • ZGetByRank
  • ZRevGetByRank
  • ZScoreRange
  • ZRevScoreRange

TODO

  • Support expiration and TTL
  • Add prefix scan and range scan for string type
  • Cli for command line use
  • Improve the performance of reopening db.
  • Improve the performance of reclaim operation.
  • Support transaction, ACID features
  • Compress the written data
  • Add cache elimination strategy (Such as LRU, LFU, Random, etc...)
  • Improve related documents

Benchmark

Benchmark Environment

  • Go version:1.14.4

  • System: macOS Catalina 10.15.7

  • CPU: 2.6GHz 6-Core Intel Core i7

  • Memory: 16 GB 2667 MHz DDR4

  • The test databases I choose:

    • Badger
    • GoLevelDB
    • Pudge

Benchmark Result

execute 100w times

go test -bench=. -benchtime=1000000x
badger 2021/05/16 21:59:53 INFO: All 0 tables opened in 0s
badger 2021/05/16 21:59:53 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 21:59:53 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    1000000             11518 ns/op            2110 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    1000000              3547 ns/op            1172 B/op         20 allocs/op
BenchmarkPutValue_GoLevelDB-12                   1000000              4659 ns/op             352 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   1000000              2838 ns/op             814 B/op         13 allocs/op
BenchmarkPutValue_Pudge-12                       1000000              8512 ns/op             791 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       1000000              1253 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            1000000              4371 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            1000000               481 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           1000000              4255 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           1000000              2986 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    46.388s

execute 250w times

go test -bench=. -benchtime=2500000x
badger 2021/05/16 22:06:08 INFO: All 0 tables opened in 0s
badger 2021/05/16 22:06:08 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 22:06:08 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    2500000             11660 ns/op            2150 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    2500000              4180 ns/op            1222 B/op         21 allocs/op
BenchmarkPutValue_GoLevelDB-12                   2500000              4637 ns/op             336 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   2500000              2942 ns/op             817 B/op         14 allocs/op
BenchmarkPutValue_Pudge-12                       2500000              9238 ns/op             763 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       2500000              1275 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            2500000              4474 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            2500000               525 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           2500000              4294 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           2500000              3038 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    119.529s

Benchmark Conclusion

Badger

Its read and wirte performance are stable. Write: 11000+ ns/op. Read: 4000+ ns/op.

GoLevelDB

Its write performance is almost 2.5x faster than Badger, and its read performance is almost 3000 ns/op, a little faster than Badger.

Pudge

Its write performance is between GoLevelDB and Badger, almost 8500 ns/op, slower than GoLevelDB. Its read performance is very fast and stable, almost 2x faster than GoLevelDB.

RoseDB

Its write performance is stable, alomost the same as GoLevelDB, 2.5x faster than Badger.

In KeyValueRamMode, since the values are all in memory, so it is the fastest of all.

In KeyOnlyRamMode, it is almost the same as GoLevelDB.

Contributing

If you are intrested in contributing to rosedb, please see here: CONTRIBUTING

Contact me

If you have any questions, you can contact me by email: [email protected]

License

rosedb is licensed under the term of the MIT License

Owner
roseduan
微信公众号:roseduan写字的地方,回复 Go,你懂的。
roseduan
Comments
  • Add Decr, DecrBy, Incr, IncrBy commands

    Add Decr, DecrBy, Incr, IncrBy commands

  • 修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改更换ActiveFile时机(Modify the timing of replacing ActivaFile)

    当前实现(Now Impl)

    • 判断当前entry是否能够写入ActiveFile(Determine whether the current entry can be written to ActiveFile)
    • 如果不能则更换ActiveFile(If not, change ActiveFile)
    • 将Entry写入到新的ActiveFile中(Write Entry to new ActiveFile)

    修改(Modify)

    • 对于当前entry, 如果ActiveFile空间不足,将超出BlockSize限制将entry写入到当前ActiveFile中(For the current entry, if the ActiveFile space is insufficient, the entry will be written to the current ActiveFile beyond the BlockSize limit)
    • 采用类似懒加载方式。每次写时先检查上次写完之后ActiveFile文件是否已满, 如果满那么写之前更换ActiveFile。(Just like lazy loading, each time you write, first check whether the ActiveFile file is full after the last write is completed, if it is full, then replace the ActiveFile before writing)

    好处(Advantage)

    参照MySQL binlog处理方式,能够最大化利用空间(Imitate MySQL binlog processing method, can maximize the use of space)

  • Add SDiff command

    Add SDiff command

    • SDiff command implemented. SMembers() logic part exported as private helper method in order to use it another method(e.g., inside of the SDiff() method).
    • Tests added for SDiff().

    Signed-off-by: Gökhan Özeloğlu [email protected]

  • build problems

    build problems

    Hello,

    Using: go version go1.16 windows/amd64

    I just tried to build RoseDB, but have errors:

    C:\goprojects\src\RoseDB\test>go build cmd\server\main.go cmd\grpc_server.go:6:2: no required module provides package github.com/roseduan/rosedb/cmd/proto; to add it: go get github.com/roseduan/rosedb/cmd/proto

    C:\goprojects\src\RoseDB\test>go get github.com/roseduan/rosedb/cmd/proto go get github.com/roseduan/rosedb/cmd/proto: no matching versions for query "upgrade"

  • Sharding for P2P model?

    Sharding for P2P model?

    Hello,

    I have just come across RoseDB and think that it might be what I need for a project that I am working on.

    One thing that I am needing is to develop for a P2P solution for the database with high number of nodes and am wondering about what it might take to have simple sharding added to RoseDB so that it can do high node clustering and spread out the data across all nodes as needed?

  • Add MSetNX command

    Add MSetNX command

    Equivalent Redis command: MSetNX

    • Helper function added to check duplicate keys.
    • Tests added.
    • Example usage added.

    Signed-off-by: Gökhan Özeloğlu [email protected]

  • db_str的bug

    db_str的bug

    func TestRoseDB_GetSet(t *testing.T) {
    	t.Run("1", func(t *testing.T) {
    		err := roseDB.Set(123, 456)
    		assert.Empty(t, err)
    
    		var res int
    		err = roseDB.GetSet(123, 567, &res)
    		assert.Empty(t, err)
    		assert.Equal(t, res, 456)
    
    		var r2 int
    		err = roseDB.Get(123, &r2)
    		assert.Empty(t, err)
    		assert.Equal(t, r2, 567) // will raise a bug r2 = 456 not 567
    	})
    
    	t.Run("2", func(t *testing.T) {
    		var res interface{}
    		err := roseDB.GetSet(123, 222, &res)
    		assert.Equal(t, err, nil)
    	})
    }
    beacause for r2 get. it get data from cache. but 567 not write into cache
    this bug i can fit it.
  • About support index mode[KeyValueMemMode / KeyOnlyMemMode] -> hash data structure support.

    About support index mode[KeyValueMemMode / KeyOnlyMemMode] -> hash data structure support.

    #60 关于这个issue提到的代码修改,此次更新关于hash部分的索引支持idxMode的设计:

    1. 如果是KeyValueMemMode, Value会被存储在索引部分,读取的时候直接从内存中读取,
    2. 如果是KeyOnlyMemMode, Value存储在磁盘中,内存中只有对应的fileId,offset等信息,可以通过这些信息从磁盘中读取对应的value。
  • Add MSet command

    Add MSet command

    Related to https://github.com/flower-corp/rosedb/issues/98.

    • MSet() command added.
    • New error added. The error message is copied from the Redis MSet command error message.
    • Tests added.
    • Example usage added.

    Signed-off-by: Gökhan Özeloğlu [email protected]

  • mmap写文件,每个文件只有一个entry

    mmap写文件,每个文件只有一个entry

    bug现象:

    使用mmap方式set一个kv,在0.data.str文件中会写入一个二进制entry;再次使用mmap set一个kv,该kv会写入1.data.str文件的开头,而不是继续在0.data.str中继续写入。 这导致16mb的0.data.str文件中,只有一个entry,剩余绝大部分空间都没有使用。并且每set一次不同的k,就会多一个文件。

    bug原因:

    文件offset数值错误。这发生在NewDBFile方法中的mmap逻辑中: 一开始我们让offset相当于文件的大小,但其实Truncate会将文件以'\0'填满,所以对于mmap来说,offset永远指的是文件固定大小,而不是实际entry要存储的位置。 由于存储逻辑中,会根据offset判断文件容量达到上限创建新文件存储entry,这就导致set新的k,会创建新的文件。

    stat, err := file.Stat()
    	if err != nil {
    		return nil, err
    	}
    
    	df := &DBFile{Id: fileId, Path: path, Offset: stat.Size(), method: method}
    
    	if method == FileIO {
    		df.File = file
    	} else {
    		if err = file.Truncate(blockSize); err != nil {
    			return nil, err
    		}
    		m, err := mmap.Map(file, os.O_RDWR, 0)
    		if err != nil {
    			return nil, err
    		}
    		df.mmap = m
    	}
    
  • 优化Open函数加载索引方式,变更store函数中切换文件处理方式

    优化Open函数加载索引方式,变更store函数中切换文件处理方式

    Open中索引加载方式

    目前实现

    对于一个存档file, 基于config的BlockSize分批次读取文件中的entry

    可能存在的问题

    如果当前用户指定了新的config,而且新配置的BlockSize(NewBlockSize)和存档file生成时的BlockSize(GenBlockSize)不同:

    • NewBlockSize < GenBlockSize: 将会导致该存档文件NewBlockSize偏移处开始的数据无法加载到索引中
    • NewBlockSize < GenBlockSize: 触发EOF

    解决方法

    修改之前分批读取方式, 一次性将文件读取到buff中,不根据配置中的BlockSize分批次读取。同时也能有效减少io次数

    变更store函数中切换文件处理方式

    目前处理方式

    先判断当前entry是否能够放到文件中,如果不能则切换新文件;然后再执行write

    可能的问题

    会存在空间浪费,尤其对于一个较大的entry而言,可能当前文件仅仅比大entry小一点,但是剩余的空间无法使用

    解决方法

    类似MySQL binlog对于事务的处理方式,当事务大小超过binlog file限制时,允许扩大binlog file以容纳最后一个事务binlog。从而节省空间

  • why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    For structs in the project, such as "listIndex or hashIndex", mu is defined as the pointer type *sync. RWMutex, in the project mu is defined as is there anything wrong with sync.RWMutex(e.g. sync. RWMutex copy problem)?

An embedded key/value database for Go.

bbolt bbolt is a fork of Ben Johnson's Bolt key/value store. The purpose of this fork is to provide the Go community with an active maintenance and de

May 10, 2022
An embedded, hardened key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

Nov 4, 2021
Hard Disk Database based on a former database

Hard Disk Database based on a former database

Nov 1, 2021
Embedded key-value store for read-heavy workloads written in Go
Embedded key-value store for read-heavy workloads written in Go

Pogreb Pogreb is an embedded key-value store for read-heavy workloads written in Go. Key characteristics 100% Go. Optimized for fast random lookups an

May 10, 2022
Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures.

Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures. capabilities which owl provides include Process approval、sql Audit、sql execute and execute as crontab、data backup and recover .

Feb 23, 2022
Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on commands and key-regexes written by GO

Welcome to NIPO Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on command

Apr 23, 2021
Beerus-DB: a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic operations

Beerus-DB · Beerus-DB is a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic

Mar 5, 2022
This is a simple graph database in SQLite, inspired by "SQLite as a document database".

About This is a simple graph database in SQLite, inspired by "SQLite as a document database". Structure The schema consists of just two structures: No

May 9, 2022
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Nov 13, 2021
Fast specialized time-series database for IoT, real-time internet connected devices and AI analytics.
Fast specialized time-series database for IoT, real-time internet connected devices and AI analytics.

unitdb Unitdb is blazing fast specialized time-series database for microservices, IoT, and realtime internet connected devices. As Unitdb satisfy the

Mar 29, 2022
VictoriaMetrics: fast, cost-effective monitoring solution and time series database
VictoriaMetrics: fast, cost-effective monitoring solution and time series database

VictoriaMetrics VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. It is available in binary release

May 14, 2022
BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go
BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go

BadgerDB BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast,

Dec 10, 2021
Fast Database engine in Go.

gaeadb gaeadb is a pure Go Database engine designed by nnsgmsone. The goal of the project is to provide a database engine for table or other complex d

Oct 29, 2021
EliasDB a graph-based database.
EliasDB a graph-based database.

EliasDB EliasDB is a graph-based database which aims to provide a lightweight solution for projects which want to store their data as a graph. Feature

May 8, 2022
Lightweight RESTful database engine based on stack data structures
Lightweight RESTful database engine based on stack data structures

piladb [pee-lah-dee-bee]. pila means stack or battery in Spanish. piladb is a lightweight RESTful database engine based on stack data structures. Crea

Mar 21, 2022
Time Series Database based on Cassandra with Prometheus remote read/write support

SquirrelDB SquirrelDB is a scalable high-available timeseries database (TSDB) compatible with Prometheus remote storage. SquirrelDB store data in Cass

Apr 25, 2022
Rk-db - Enterprise level database bootstrapper with YAML based on rk-entry in Golang

rk-db Enterprise level database bootstrapper with YAML in golang. This belongs to rk-boot family. We suggest use this lib from rk-boot. Database Statu

Mar 7, 2022
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

May 12, 2022
The Prometheus monitoring system and time series database.

Prometheus Visit prometheus.io for the full documentation, examples and guides. Prometheus, a Cloud Native Computing Foundation project, is a systems

May 14, 2022