Distributed disk storage database based on Raft and Redis protocol.

IceFireDB

Last update: Dec 31, 2022

Comments: 11

IceFireDB

Distributed disk storage system based on Raft and RESP protocol.

High performance
Distributed consistency
Reliable LSM disk storage
Cold and hot mixed data storage structure
Multiple groups of raft clusters, horizontal expansion and contraction of data groups（Upgrading soon）
More advanced cache implementation, faster LSM persistent storage

Imagine Architecture

Command support

1. String operating

APPEND
BITCOUNT
BITOP
BITPOS
DECR
DECRBY
DEL
EXISTS
GET
GETBIT
SETBIT
GETRANGE
GETSET
INCR
INCRBY
MGET
MSET
SET
SETEX
SETEXAT
SETRANGE
EXPIRE
EXPIREAT
TTL

2. Hash operating

HSET
HGET
HDEL
HEXISTS
HGETALL
HINCRBY
HKEYS
HLEN
HMGET
HMSET
HSETEX
HSTRLEN
HVALS
HCLEAR
HMCLEAR
HEXPIRE
HEXPIREAT
HKEYEXIST
HTTL

3. List operating

RPUSH
LPOP
LINDEX
LPUSH
RPOP
LRANGE
LSET
LLEN
RPOPLPUSH
LCLEAR
LMCLEAR
LEXPIRE
LEXPIREAT
LKEYEXISTS
LTRIM
LTTL

4. Sorted Sets

ZADD
ZCARD
ZCOUNT
ZREM
ZCLEAR
ZRANK
ZRANGE
ZREVRANGE
ZSCORE
ZINCRBY
ZREVRANK
ZRANGEBYSCORE
ZREVRANGEBYSCORE
ZREMRANGEBYSCORE
ZREMRANGEBYRANK

5. Sets

SADD
SCARD
SDIFF
SDIFFSTORE
SINTER
SINTERSTORE
SISMEMBER
SMEMBERS
SREM
SUNION
SUNIONSTORE
SCLEAR
SMCLEAR
SEXPIRE
SEXPIREAT
STTL
SPERSIST
SKEYEXISTS

6. System cmd

INFO
FLUSHALL
HSCAN
SSCAN
ZSCAN
XSCAN
XHSCAN
XSSCAN
XZSCAN

Performance

corerman@ubuntu:~/DATA/ICODE/GoLang/IceFireDB$ redis-benchmark  -h 127.0.0.1 -p 11001 -n 10000000 -t set,get -c 512 -P 512 -q

SET: 253232.12 requests per second
GET: 2130875.50 requests per second

License

Thanks

I stood on the shoulders of giants and did only simple things. Thank you for your attention.

Disclaimers

When you use this software, you have agreed and stated that the author, maintainer and contributor of this software are not responsible for any risks, costs or problems you encounter. If you find a software defect or BUG, please submit a patch to help improve it!

Owner

IceFireDB

Distributed disk storage database based on Raft and Redis protocol.

https://github.com/IceFireDB/IceFireDB https://www.icefiredb.com

Comments

chore(deps): bump github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2
Bumps github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2.

Release notes

Sourced from github.com/dgraph-io/badger/v3's releases.

BadgerDB v3.2103.2

This patch release contains:

Fixed

fix(compact): close vlog after the compaction at L0 has been completed (#1752)

fix(builder): put the upper limit on reallocation (#1748)

deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)

fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)

fix(pubsub): avoid deadlock in publisher and subscriber (#1749) (#1751)

Full Changelog: https://github.com/dgraph-io/badger/compare/v3.2103.1...v3.2103.2

Changelog

Sourced from github.com/dgraph-io/badger/v3's changelog.

[3.2103.2] - 2021-10-07

Fixed

fix(compact): close vlog after the compaction at L0 has completed (#1752)

fix(builder): put the upper limit on reallocation (#1748)

deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)

fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)

fix(pubsub): avoid deadlock in publisher and subscriber (#1749) (#1751)

Commits

f388e91 Update changelog for v3.2103.2

09a6157 fix(compact): close vlog after the compaction at L0 has completed (#1752)

4298c58 fix(builder): put the upper limit on reallocation (#1748)

644971a deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)

0c13869 Fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)

e4e2ad7 fix: avoid deadlock in publisher and subscriber (#1749) (#1751)

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot will merge this PR once CI passes on it, as requested by @gitsrc.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
chore(deps): bump github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4
Bumps github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4.

Release notes

Sourced from github.com/go-redis/redis/v8's releases.

v8.11.4

Please refer to CHANGELOG.md for details

Changelog

Sourced from github.com/go-redis/redis/v8's changelog.

8.11.4 (2021-10-04)

Features

add acl auth support for sentinels (f66582f)

add Cmd.{String,Int,Float,Bool}Slice helpers and an example (5d3d293)

add SetVal method for each command (168981d)

v8.11

Remove OpenTelemetry metrics.

Supports more redis commands and options.

v8.10

Removed extra OpenTelemetry spans from go-redis core. Now go-redis instrumentation only adds a single span with a Redis command (instead of 4 spans). There are multiple reasons behind this decision:

Traces become smaller and less noisy.

It may be costly to process those 3 extra spans for each query.

go-redis no longer depends on OpenTelemetry.

Eventually we hope to replace the information that we no longer collect with OpenTelemetry Metrics.

v8.9

Changed PubSub.Channel to only rely on Ping result. You can now use WithChannelSize, WithChannelHealthCheckInterval, and WithChannelSendTimeout to override default settings.

v8.8

To make updating easier, extra modules now have the same version as go-redis does. That means that you need to update your imports:

github.com/go-redis/redis/extra/redisotel -> github.com/go-redis/redis/extra/redisotel/v8 github.com/go-redis/redis/extra/rediscensus -> github.com/go-redis/redis/extra/rediscensus/v8

v8.5

knadh contributed long-awaited ability to scan Redis Hash into a struct:

</tr></table>

... (truncated)

Commits

fa51524 Merge pull request #1917 from go-redis/release/v8.11.4

0457d0a chore: release v8.11.4 (release.sh)

6dcfc56 chore: use conv commit message

86258a1 chore: lazy reload when moved or ask

98bb99d Fix Redis Cluster issue during roll outs of new nodes with same addr (#1914)

5072031 chore: add Lua scripting example

739c188 chore: tweak GH actions to run all jobs

cc09f96 Merge pull request #1909 from go-redis/feature/slice-helpers

5d3d293 feat: add Cmd.{String,Int,Float,Bool}Slice helpers and an example

329d87f Merge pull request #1906 from Stavrospanakakis/master

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot will merge this PR once it's up-to-date and CI passes on it, as requested by @gitsrc.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Feat add crdt driver

New decentralized crdt kV database driver.

Usage:

IceFireDB -a :11001 \
    --storage-backend crdt \
    --servicename icefiredb \
    --nettopic icefiredb-net \
    --datatopic icefiredb-data

Documentation to start up and query cluster?

Hello,

I was able to compile IceFireDB but have not found any documentation that shows the steps to start up a multi-node cluster and then make some queries?

I am interested in highly distributed P2P usage and scaling limits of IceFireDB Thanks
Missing Copyright Notice and License to Uhaha

Hi,

The source code in the rafthub directory of this project is directly copied from https://github.com/tidwall/uhaha/blob/master/uhaha.go, which is 3K+ lines of original source code written and copyrighted by me.

The original copyright notice and LICENSE were omitted.

The Uhaha LICENSE states:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
chore(deps): bump github.com/tidwall/redcon from 1.4.1 to 1.4.2
Bumps github.com/tidwall/redcon from 1.4.1 to 1.4.2.

Commits

babe351 Upgrade modules

4d3824a Create FUNDING.yml

0cb26bc Delete FUNDING.yml

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
feat(RedisCmdRewrite): Implementation of redis instruction rewrite
Hello, because the raft log rollback exception occurs when raft is combined with the underlying storage, I helped you build an instruction rewrite plug-in layer, and "ArgsHandler" implements the redis instruction rewrite interface. Processing logic introduction:

Receive the redis command and current time of type [][]string.

Traverse the args, and rewrite the instructions as setex, expire, lexpire, and hexpire through the timestamp.
feat(hybriddb): add the cache layer
Hello, I helped you build a hot data caching layer, "Hybriddb" implements ledisdb storage interface.

Introduction to processing logic:

When reading kv data, first check the memory cache, if it is lost, query the disk data and store it in the memory

When writing kv data, directly clear the memory cache and wait for the next read data cache to the memory
feat: add tests for cmds
Hi,

Great works, I add tests for commands.

~~Still working in progress.~~

Done!

[x] strings commands

[x] hash commands

[x] list commands

[x] sorted commands
Neat idea, but need more documentation

I want to understand what is the architecture. What is the process of configuring, adding, removing nodes?

How was raft integrated with redis, IE. where was raft end and redis begin?
chore(deps): bump github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0 in /IceFireDB-PubSub
Bumps github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0.

Release notes

Sourced from github.com/libp2p/go-libp2p's releases.

v0.18.0

We're happy to announce the v0.18.0 release of go-libp2p.

🛠 Breaking Changes

In this release, we're continuing our repo consolidation. We've moved the following repositories into go-libp2p:

go-libp2p-autonat

go-libp2p-discovery

go-libp2p-connmgr

go-addr-utils

go-libp2p-netutil

go-sockaddr

If you've been using functions from these repositories, please make sure to point those to go-libp2p.

🔦 Highlights

The main feature of v0.18.0 is the Resource Manager. The Resource Manager allows the configuration of (among others):

the number of incoming / outgoing connections

the number of incoming / outgoing streams

the number of streams per protocol, and per service

memory usage

For details on how this works, and how to change limits, refer to the README of the Resource Manager.

In this release, we've finished the transition from ipfs/go-log@v1 to ipfs/go-log@v2. One less dependency!

Changelog

github.com/libp2p/go-libp2p:

update go-libp2p-resource manager, release v0.18.0 (#1361) (libp2p/go-libp2p#1361)

fix flaky BackoffConnector test (#1353) (libp2p/go-libp2p#1353)

release v0.18.0-rc6 (#1350) (libp2p/go-libp2p#1350)

update mplex (#1348) (libp2p/go-libp2p#1348)

release v0.18.0-rc5 (libp2p/go-libp2p#1341)

update README (#1330) (libp2p/go-libp2p#1330)

fix parsing of IP addresses for zeroconf initialization (#1338) (libp2p/go-libp2p#1338)

fix flaky TestBackoffConnector test (#1328) (libp2p/go-libp2p#1328)

release v0.18.0-rc4 (libp2p/go-libp2p#1327)

update go-libp2p-yamux and go-libp2p-quic-transport (#1326) (libp2p/go-libp2p#1326)

fix (and speed up) flaky TestBackoffConnector test (#1316) (libp2p/go-libp2p#1316)

increase the UDP receiver buffer size on CI (#1323) (libp2p/go-libp2p#1323)

fix flaky TestAutoRelay test (#1322) (libp2p/go-libp2p#1322)

deflake resource manager tests, take 2 (libp2p/go-libp2p#1318)

fix race condition causing TestAutoNATServiceDialError test failure (#1312) (libp2p/go-libp2p#1312)

disable flaky relay example test on CI (#1219) (libp2p/go-libp2p#1219)

fix flaky resource manager tests (libp2p/go-libp2p#1315)

update deps, fixing nil peer scope pointer issues in connection upgrading (libp2p/go-libp2p#1309)

... (truncated)

Commits

5bab581 update go-libp2p-resource manager, release v0.18.0 (#1361)

13d0381 fix flaky BackoffConnector test (#1353)

f0db9cc release v0.18.0-rc6 (#1350)

49d75d7 update mplex (#1348)

4c0e2b8 Merge pull request #1341 from libp2p/release-v0.18.0-rc5

2d8f0a8 bump version to v0.18.0-rc5

32d7977 update go-libp2p-swarm and go-tcp-transport

4b236dd update README (#1330)

49479c7 fix parsing of IP addresses for zeroconf initialization (#1338)

ccdd17f fix flaky TestBackoffConnector test (#1328)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page.

Raft library Raft is a protocol with which a cluster of nodes can maintain a replicated state machine.

Raft library Raft is a protocol with which a cluster of nodes can maintain a replicated state machine. The state machine is kept in sync through the u

Oct 15, 2021

6.824-Raft - There are three roles in Raft algorithm, Follower, Candidate, Leader, each node store currentTerm, votedFor and log

6.824-Raft Raft note There are three roles in Raft algorithm, Follower, Candidat

Feb 3, 2022

A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

Dec 31, 2022

A linearizability distributed database by raft and wisckey.

AlfheimDB A linearizability distributed database by raft and wisckey, which supports redis client. Build This project build by mage, you will need ins

Jul 18, 2022

The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Nov 19, 2021

Golang implementation of the Raft consensus protocol

raft raft is a Go library that manages a replicated log and can be used with an FSM to manage replicated state machines. It is a library for providing

Jan 9, 2023

An implementation of a distributed KV store backed by Raft tolerant of node failures and network partitions 🚣

barge A simple implementation of a consistent, distributed Key:Value store which uses the Raft Concensus Algorithm. This project launches a cluster of

Nov 24, 2021

Easy to use Raft library to make your app distributed, highly available and fault-tolerant

An easy to use customizable library to make your Go application Distributed, Highly available, Fault Tolerant etc... using Hashicorp's Raft library wh

Nov 16, 2022

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Jan 1, 2023

Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Dec 24, 2019

Distributed disk storage database based on Raft and Redis protocol.

IceFireDB

Imagine Architecture

Command support

1. String operating

2. Hash operating

3. List operating

4. Sorted Sets

5. Sets

6. System cmd

Performance

License

Thanks

I stood on the shoulders of giants and did only simple things. Thank you for your attention.

Disclaimers

Owner

IceFireDB

Comments

chore(deps): bump github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2

BadgerDB v3.2103.2

Fixed

[3.2103.2] - 2021-10-07

Fixed

chore(deps): bump github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4

v8.11.4

8.11.4 (2021-10-04)

Features

v8.11

v8.10

v8.9

v8.8

v8.5

Feat add crdt driver

Documentation to start up and query cluster?

Missing Copyright Notice and License to Uhaha

chore(deps): bump github.com/tidwall/redcon from 1.4.1 to 1.4.2

feat(RedisCmdRewrite): Implementation of redis instruction rewrite

feat(hybriddb): add the cache layer

feat: add tests for cmds

Neat idea, but need more documentation

chore(deps): bump github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0 in /IceFireDB-PubSub

v0.18.0

🛠 Breaking Changes

🔦 Highlights

Changelog

Related tags

Raft library Raft is a protocol with which a cluster of nodes can maintain a replicated state machine.

6.824-Raft - There are three roles in Raft algorithm, Follower, Candidate, Leader, each node store currentTerm, votedFor and log

A distributed MySQL binlog storage system built on Raft

A linearizability distributed database by raft and wisckey.

The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

Golang implementation of the Raft consensus protocol

An implementation of a distributed KV store backed by Raft tolerant of node failures and network partitions 🚣

Easy to use Raft library to make your app distributed, highly available and fault-tolerant

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

A feature complete and high performance multi-group Raft library in Go.

BlobStore is a highly reliable,highly available and ultra-large scale distributed storage system

Simplified distributed locking implementation using Redis

A naive implementation of Raft consensus algorithm.

Raft: a consensus algorithm for managing a replicated log

This is my implementation of Raft consensus algorithm that I did for own learning.

A distributed key-value storage system developed by Alibaba Group

Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.