decentralized kv store based on pubsub in libp2p

pointers

decentralized kv store based on pubsub in libp2p

Protocol Specification

Over View

The pointer would receive updates from pubsub network and broadcast their local latest data so that the network would reach a final consistency.

New nodes who join the network will connect to nearby nodes and ask them for data.

Operations Message

To put, modify or delete a value from the network, a node would publish an Operate message.

enum Operations {
  Mut = 0;
  Del = 1;
}

message Operate {
  Operations op = 1;
  optional string value = 2;

  string sign = 3;
  int64 time = 4;
}

Message Operate includes the type of operation in the form of enum, the optional new value, the signature of the publisher, and the time it sent.

Nodes would first compare the time of the message with the time in existed local storage. If the message is later than that in storage, nodes would start to check the validity of the signature and if it is valid, the local storage would be updated by the new message.

Mut in Operations would update the value, and Del would wipe out this value in the whole network. Every node that receives the Del operation would delete the corresponding value and end the subscription of the corresponding topic in pubsub.

Ask & Answer Message

To get a value from nodes it connected, a node would send an Ask message to and receive Answer from them.

message Ask {
  string key = 1;
}

enum Status {
  Ok = 0;
  NotExist = 1;
}

message Answer {
  Status status = 1;
  optional Operate op = 2;
}

Message Ask would tell other nodes that the value of a specific key is needed. Any node that received this message could check their storage to see if they have this value.

When a node finds the value, it would return an Answer message with the status field that tells whether it found the corresponding value or not. If status is Ok, the receiver should be able to read the op field and store this value in its local storage.

Storage

In the need of validating the message, the value in the data store should not be only the single value, so there are other data other than the value.

message StoreValue {
    string value = 2;

    string sign = 3;
    int64 time = 4;
}

The message StoreValue would be stored in the local key-value database in the form of binary.

Mechanism Details

A node can store and listen to any data it is interested, so every key-value has its corresponding pubsub topic. The name of the topic would be generated from:

Topic := "/pointer/record/" + Key

And then the node would listen to the topic, getting ready to update its local value whenever.

If it is new to this value, it is still possible to get quick update from nearby nodes. The node will discover some nodes with the same topic, and send Ask message to them with a protocol id from:

ProtocolId := "/pointer/fetch/0.0.1"

Others would send their local value to the new one, and the new one is able to catch up with the network.

Every node in the same topic will send an Operate message to the network with a specific time interval, enabling others to catch up with new values.

Similar Resources

ShockV is a simple key-value store with RESTful API

ShockV is a simple key-value store with RESTful API

ShockV is a simple key-value store based on badgerDB with RESTful API. It's best suited for experimental project which you need a lightweight data store.

Sep 26, 2021

A rest-api that works with golang as an in-memory key value store

In Store A rest-api that works with golang as an in-memory key value store Usage Fist of all, clone the repo with the command below. You must have gol

Oct 24, 2021

simple kv store engine inspired by bitcask

tiny-bitcask simple kv store engine inspired by bitcask Inspired By BitCask BitCask System Design DB启动时,系统内存储着merged-data-file 与 hint-file 系统通过读取hint-

Nov 14, 2021

Distributed key-value store

Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Nov 15, 2021

Simple in memory key-value store.

Simple in memory key-value store. Development This project is written in Go. Make sure you have Go installed (download). Version 1.17 or higher is req

Nov 6, 2021

A simple in-memory key-value store application

A simple in-memory key-value store application

vtec vtec, is a simple in-memory key-value store application. vtec provides persistence by appending transactions to a json file and restoring data fr

Jun 22, 2022

ApolloKV is a lightweight KV store written in GoLang

ApolloKV is a lightweight KV store written in GoLang

Dec 2, 2021

Biscuit is a multi-region HA key-value store for your AWS infrastructure secrets.

Biscuit Biscuit is a simple key-value store for your infrastructure secrets. Is Biscuit right for me? Biscuit is most useful to teams already using AW

Nov 10, 2022

An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

go-cache go-cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major

Jan 3, 2023
Comments
  • how to pub one host, and fetch it on the other host?

    how to pub one host, and fetch it on the other host?

    If there are many nodes on different machine, one node pub data, how to fetch data from other nodes? If pub node goes offline, other nodes can fetch the data?

CrankDB is an ultra fast and very lightweight Key Value based Document Store.

CrankDB is a ultra fast, extreme lightweight Key Value based Document Store.

Apr 12, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Jan 1, 2023
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

Dec 28, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Jan 4, 2023
a persistent real-time key-value store, with the same redis protocol with powerful features
a persistent real-time key-value store, with the same redis protocol with powerful features

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Dec 25, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jan 6, 2023
Pogreb is an embedded key-value store for read-heavy workloads written in Go.
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Dec 29, 2022
High-performance, columnar, in-memory store with bitmap indexing in Go
High-performance, columnar, in-memory store with bitmap indexing in Go

This package contains a high-performance, columnar, in-memory storage engine that supports fast querying, update and iteration with zero-allocations and bitmap indexing.

Jan 8, 2023
yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.
yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.

yakv (yak-v. (originally intended to be "yet-another-key-value store")) is a simple, in-memory, concurrency-safe key-value store for hobbyists. yakv provides persistence by appending transactions to a transaction log and restoring data from the transaction log on startup.

Feb 24, 2022
Multithreaded key value pair store using thread safe locking mechanism allowing concurrent reads
Multithreaded key value pair store using thread safe locking mechanism allowing concurrent reads

Project Amnesia A Multi-threaded key-value pair store using thread safe locking mechanism allowing concurrent reads. Curious to Try it out?? Check out

Oct 29, 2022