Distributed disk storage database based on Raft and Redis protocol.

IceFireDB

IceFireDB

test build FOSSA Status

Distributed disk storage system based on Raft and RESP protocol.

  1. High performance
  2. Distributed consistency
  3. Reliable LSM disk storage
  4. Cold and hot mixed data storage structure
  5. Multiple groups of raft clusters, horizontal expansion and contraction of data groups（Upgrading soon）
  6. More advanced cache implementation, faster LSM persistent storage

Imagine Architecture

IceFireDB_Architecture

Command support

1. String operating

  • APPEND
  • BITCOUNT
  • BITOP
  • BITPOS
  • DECR
  • DECRBY
  • DEL
  • EXISTS
  • GET
  • GETBIT
  • SETBIT
  • GETRANGE
  • GETSET
  • INCR
  • INCRBY
  • MGET
  • MSET
  • SET
  • SETEX
  • SETEXAT
  • SETRANGE
  • EXPIRE
  • EXPIREAT
  • TTL

2. Hash operating

  • HSET
  • HGET
  • HDEL
  • HEXISTS
  • HGETALL
  • HINCRBY
  • HKEYS
  • HLEN
  • HMGET
  • HMSET
  • HSETEX
  • HSTRLEN
  • HVALS
  • HCLEAR
  • HMCLEAR
  • HEXPIRE
  • HEXPIREAT
  • HKEYEXIST
  • HTTL

3. List operating

  • RPUSH
  • LPOP
  • LINDEX
  • LPUSH
  • RPOP
  • LRANGE
  • LSET
  • LLEN
  • RPOPLPUSH
  • LCLEAR
  • LMCLEAR
  • LEXPIRE
  • LEXPIREAT
  • LKEYEXISTS
  • LTRIM
  • LTTL

4. Sorted Sets

  • ZADD
  • ZCARD
  • ZCOUNT
  • ZREM
  • ZCLEAR
  • ZRANK
  • ZRANGE
  • ZREVRANGE
  • ZSCORE
  • ZINCRBY
  • ZREVRANK
  • ZRANGEBYSCORE
  • ZREVRANGEBYSCORE
  • ZREMRANGEBYSCORE
  • ZREMRANGEBYRANK

5. Sets

  • SADD
  • SCARD
  • SDIFF
  • SDIFFSTORE
  • SINTER
  • SINTERSTORE
  • SISMEMBER
  • SMEMBERS
  • SREM
  • SUNION
  • SUNIONSTORE
  • SCLEAR
  • SMCLEAR
  • SEXPIRE
  • SEXPIREAT
  • STTL
  • SPERSIST
  • SKEYEXISTS

6. System cmd

  • INFO
  • FLUSHALL
  • HSCAN
  • SSCAN
  • ZSCAN
  • XSCAN
  • XHSCAN
  • XSSCAN
  • XZSCAN

Performance

corerman@ubuntu:~/DATA/ICODE/GoLang/IceFireDB$ redis-benchmark  -h 127.0.0.1 -p 11001 -n 10000000 -t set,get -c 512 -P 512 -q

SET: 253232.12 requests per second
GET: 2130875.50 requests per second

License

FOSSA Status

Thanks

I stood on the shoulders of giants and did only simple things. Thank you for your attention.

Disclaimers

When you use this software, you have agreed and stated that the author, maintainer and contributor of this software are not responsible for any risks, costs or problems you encounter. If you find a software defect or BUG, ​​please submit a patch to help improve it!

Owner
IceFireDB
Distributed disk storage database based on Raft and Redis protocol.
IceFireDB
https://github.com/IceFireDB/IceFireDB https://www.icefiredb.com
Comments
  • chore(deps): bump github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2

    chore(deps): bump github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2

    Bumps github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2.

    Release notes

    Sourced from github.com/dgraph-io/badger/v3's releases.

    BadgerDB v3.2103.2

    This patch release contains:

    Fixed

    • fix(compact): close vlog after the compaction at L0 has been completed (#1752)
    • fix(builder): put the upper limit on reallocation (#1748)
    • deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
    • fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
    • fix(pubsub): avoid deadlock in publisher and subscriber (#1749) (#1751)

    Full Changelog: https://github.com/dgraph-io/badger/compare/v3.2103.1...v3.2103.2

    Changelog

    Sourced from github.com/dgraph-io/badger/v3's changelog.

    [3.2103.2] - 2021-10-07

    Fixed

    • fix(compact): close vlog after the compaction at L0 has completed (#1752)
    • fix(builder): put the upper limit on reallocation (#1748)
    • deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
    • fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
    • fix(pubsub): avoid deadlock in publisher and subscriber (#1749) (#1751)
    Commits
    • f388e91 Update changelog for v3.2103.2
    • 09a6157 fix(compact): close vlog after the compaction at L0 has completed (#1752)
    • 4298c58 fix(builder): put the upper limit on reallocation (#1748)
    • 644971a deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
    • 0c13869 Fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
    • e4e2ad7 fix: avoid deadlock in publisher and subscriber (#1749) (#1751)
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot will merge this PR once CI passes on it, as requested by @gitsrc.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • chore(deps): bump github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4

    chore(deps): bump github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4

    Bumps github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4.

    Release notes

    Sourced from github.com/go-redis/redis/v8's releases.

    v8.11.4

    Please refer to CHANGELOG.md for details

    Changelog

    Sourced from github.com/go-redis/redis/v8's changelog.

    8.11.4 (2021-10-04)

    Features

    • add acl auth support for sentinels (f66582f)
    • add Cmd.{String,Int,Float,Bool}Slice helpers and an example (5d3d293)
    • add SetVal method for each command (168981d)

    v8.11

    • Remove OpenTelemetry metrics.
    • Supports more redis commands and options.

    v8.10

    • Removed extra OpenTelemetry spans from go-redis core. Now go-redis instrumentation only adds a single span with a Redis command (instead of 4 spans). There are multiple reasons behind this decision:

      • Traces become smaller and less noisy.
      • It may be costly to process those 3 extra spans for each query.
      • go-redis no longer depends on OpenTelemetry.

      Eventually we hope to replace the information that we no longer collect with OpenTelemetry Metrics.

    v8.9

    • Changed PubSub.Channel to only rely on Ping result. You can now use WithChannelSize, WithChannelHealthCheckInterval, and WithChannelSendTimeout to override default settings.

    v8.8

    • To make updating easier, extra modules now have the same version as go-redis does. That means that you need to update your imports:
    github.com/go-redis/redis/extra/redisotel -> github.com/go-redis/redis/extra/redisotel/v8
github.com/go-redis/redis/extra/rediscensus -> github.com/go-redis/redis/extra/rediscensus/v8

    v8.5

    • knadh contributed long-awaited ability to scan Redis Hash into a struct:
    </tr></table>

    ... (truncated)

    Commits
    • fa51524 Merge pull request #1917 from go-redis/release/v8.11.4
    • 0457d0a chore: release v8.11.4 (release.sh)
    • 6dcfc56 chore: use conv commit message
    • 86258a1 chore: lazy reload when moved or ask
    • 98bb99d Fix Redis Cluster issue during roll outs of new nodes with same addr (#1914)
    • 5072031 chore: add Lua scripting example
    • 739c188 chore: tweak GH actions to run all jobs
    • cc09f96 Merge pull request #1909 from go-redis/feature/slice-helpers
    • 5d3d293 feat: add Cmd.{String,Int,Float,Bool}Slice helpers and an example
    • 329d87f Merge pull request #1906 from Stavrospanakakis/master
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot will merge this PR once it's up-to-date and CI passes on it, as requested by @gitsrc.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Feat add crdt driver

    Feat add crdt driver

    New decentralized crdt kV database driver.

    Usage:

    IceFireDB -a :11001 \
    --storage-backend crdt \
    --servicename icefiredb \
    --nettopic icefiredb-net \
    --datatopic icefiredb-data
  • Documentation to start up and query cluster?

    Documentation to start up and query cluster?

    Hello,

    I was able to compile IceFireDB but have not found any documentation that shows the steps to start up a multi-node cluster and then make some queries?

    I am interested in highly distributed P2P usage and scaling limits of IceFireDB Thanks

  • Missing Copyright Notice and License to Uhaha

    Missing Copyright Notice and License to Uhaha

    Hi,

    The source code in the rafthub directory of this project is directly copied from https://github.com/tidwall/uhaha/blob/master/uhaha.go, which is 3K+ lines of original source code written and copyrighted by me.

    The original copyright notice and LICENSE were omitted.

    The Uhaha LICENSE states:

    The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

  • chore(deps): bump github.com/tidwall/redcon from 1.4.1 to 1.4.2

    chore(deps): bump github.com/tidwall/redcon from 1.4.1 to 1.4.2

    Bumps github.com/tidwall/redcon from 1.4.1 to 1.4.2.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • feat(RedisCmdRewrite): Implementation of redis instruction rewrite

    feat(RedisCmdRewrite): Implementation of redis instruction rewrite

    Hello, because the raft log rollback exception occurs when raft is combined with the underlying storage, I helped you build an instruction rewrite plug-in layer, and "ArgsHandler" implements the redis instruction rewrite interface. Processing logic introduction:

    • Receive the redis command and current time of type [][]string.
    • Traverse the args, and rewrite the instructions as setex, expire, lexpire, and hexpire through the timestamp.
  • feat(hybriddb): add the cache layer

    feat(hybriddb): add the cache layer

    Hello, I helped you build a hot data caching layer, "Hybriddb" implements ledisdb storage interface.

    Introduction to processing logic:

    • When reading kv data, first check the memory cache, if it is lost, query the disk data and store it in the memory

    • When writing kv data, directly clear the memory cache and wait for the next read data cache to the memory

  • feat: add tests for cmds

    feat: add tests for cmds

    Hi,

    Great works, I add tests for commands.

    Still working in progress.

    Done!

    • [x] strings commands
    • [x] hash commands
    • [x] list commands
    • [x] sorted commands
  • Neat idea, but need more documentation

    Neat idea, but need more documentation

    I want to understand what is the architecture. What is the process of configuring, adding, removing nodes?

    How was raft integrated with redis, IE. where was raft end and redis begin?

  • chore(deps): bump github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0 in /IceFireDB-PubSub

    chore(deps): bump github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0 in /IceFireDB-PubSub

    Bumps github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0.

    Release notes

    Sourced from github.com/libp2p/go-libp2p's releases.

    v0.18.0

    We're happy to announce the v0.18.0 release of go-libp2p.

    🛠 Breaking Changes

    In this release, we're continuing our repo consolidation. We've moved the following repositories into go-libp2p:

    • go-libp2p-autonat
    • go-libp2p-discovery
    • go-libp2p-connmgr
    • go-addr-utils
    • go-libp2p-netutil
    • go-sockaddr

    If you've been using functions from these repositories, please make sure to point those to go-libp2p.

    🔦 Highlights

    The main feature of v0.18.0 is the Resource Manager. The Resource Manager allows the configuration of (among others):

    • the number of incoming / outgoing connections
    • the number of incoming / outgoing streams
    • the number of streams per protocol, and per service
    • memory usage

    For details on how this works, and how to change limits, refer to the README of the Resource Manager.

    In this release, we've finished the transition from ipfs/go-log@v1 to ipfs/go-log@v2. One less dependency!

    Changelog

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page.
Related tags
Key-Value Store redis golang distributed-systems database disk nosql key-value storage high-performance cache raft leveldb persistence databases kvstore distributed-database lsm
Simple Distributed key-value database (in-memory/disk) written with Golang.

Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get

Jan 18, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Jan 4, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm
 The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Jul 26, 2022
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm
 Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the R

Dec 7, 2022
a persistent real-time key-value store, with the same redis protocol with powerful features
 a persistent real-time key-value store, with the same redis protocol with powerful features

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Dec 25, 2022
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.

Consul Website: https://www.consul.io Tutorials: HashiCorp Learn Forum: Discuss Consul is a distributed, highly available, and data center aware solut

Jan 2, 2023
Distributed, fault-tolerant key-value storage written in go.
 Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Jan 3, 2023
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Jan 1, 2023
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

Dec 28, 2022
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.
 rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.

Dec 28, 2022
Implementation of distributed key-value system based on TiKV

Distributed_key-value_system A naive implementation of distributed key-value system based on TiKV Features Features of this work are listed below: Dis

Mar 7, 2022
A key-value db api with multiple storage engines and key generation
 A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

Apr 5, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Jan 4, 2023
moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library.

Dec 18, 2022
Fault tolerant, sharded key value storage written in GoLang
 Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Nov 1, 2022
A key-value storage transaction interpretator.

kv-txn-interpreter A key-value storage transaction interpreter, which provides an etcd-like transaction interface to help you build a transaction over

Feb 22, 2022
Keyval - A simple key-value storage library written in Go

keyval keyval is a simple key-value storage library written in Go and its back c

Sep 16, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
 GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jan 6, 2023
Distributed key-value store
 Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Nov 15, 2021