500 Resources
Golang distributed-storage Libraries
Industrial IoT Messaging and Device Management Platform
Mainflux Mainflux is modern, scalable, secure, open-source, and patent-free IoT cloud platform written in Go. It accepts user and thing (sensor, actua
Instant Kubernetes-Native Application Observability
Pixie is an open source observability tool for Kubernetes applications. Use Pixie to view the high-level state of your cluster (service maps, cluster
A highly scalable and serverless unique ID generator for use in distributed systems.
A highly scalable and serverless unique ID generator for use in distributed systems. Written in GoLang. Inspired by Twitters Snowflake.
Edge Orchestration project is to implement distributed computing between Docker Container enabled devices.
Edge Orchestration Introduction The main purpose of Edge Orchestration project is to implement distributed computing between Docker Container enabled
A custom IPFS/Filecoin node that makes it easy to pin IPFS content and make Filecoin deals.
Estuary An experimental ipfs node Building Requirements: go (1.15 or higher) jq hwloc opencl Run make clean all inside the estuary directory Running T
A server for TurboRepo Remote Cache to store cache artefacts in Google Cloud Storage or Amazon S3
Tapico Turborepo Remote Cache This is an implementation of Vercel's Turborepo Remote Cache API endpoints used by the turborepo CLI command. This solut
BTFS - The First Scalable Decentralized Storage System - A Foundational Platform for Decentralized Applications
go-btfs What is BTFS? BitTorrent File System (BTFS) is a protocol forked from IPFS that utilizes the TRON network and the BitTorrent Ecosystem for int
Decentralized,distributed,peer-to-peer database.
P2PDB 中文 | English 注意: 本项目只作为新型数据库技术理论验证使用,当前不具备任何生产可用性。 简介 P2PDB(p2p数据库),一个去中心化、分布式、点对点数据库、P2PDB使用IPFS-libp2p构建分布式网络和IPFS-pubsub与对等节点同步数据。P2PDB期望打造一个
A High Performance Object Storage released under Apache License
MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag
QingStor Object Storage service support for go-storage
go-services-qingstor QingStor Object Storage service support for go-storage. Install go get github.com/minhjh/go-service-qingstor/v3 Usage import ( "
The management of multiple apps running over different ports made easy
Ergo Ergo Proxy - The reverse proxy agent for local domain management. The management of multiple apps running over different ports made easy through
💾 Wolke API is the API behind Wolke image storage and processing aswell as user management
💾 Wolke API Wolke API is the API behind Wolke image storage and processing aswell as user management Deploying To deploy Wolke Bot you'll need podman
Golang rate limiters for distributed applications
Distributed rate limiters for Golang Rate limiters for distributed applications in Golang with configurable back-ends and distributed locks. Any types
MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.
MinIO Client Quickstart Guide MinIO Client (mc) provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff, find etc. It supports f
🍔 Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result.
🍔 Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result. The server transfer request to a third-party resource for .csv-file uploading and saves the products to own database.
Simple distributed kv-store using ABD algorithm.
Distributed-kv-store Simple distributed kv-store using ABD algorithm. API GET /key Get value by key. 302 = found key. PUT /key Put key with value. 201
Retrying made simple and easy for golang :repeat:
retry-go Retrying made simple and easy for golang. Installation go get -u github.com/rafaeljesus/retry-go Usage Do package main import ( "time"
Simple Distributed key-value database (in-memory/disk) written with Golang.
Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get
A highly scalable and serverless unique ID generator for use in distributed systems. Written in GoLang. Inspired by Twitters Snowflake.
GOFLAKE A highly scalable, customizable, and serverless unique ID generator for use in distributed systems. Written in GoLang. Inspired by Twitters Sn
a wrapper around BadgerDB providing a simple API.
Carbon Cache A wrapper around BadgerDB providing a simple API. NOTE This package is provided "as is" with no guarantee. Use it at your own risk and al
csg ("Credential Storage with Go") - a tool to organize the storage of credentials found during a CTF or Pentest.
csg csg ("Credential Storage with Go") - a tool to organize the storage of credentials found during a CTF or Pentest. Check out my blog on csg for mor
Google Drive client for the commandline
drive drive is a tiny program to pull or push Google Drive files. drive was originally developed by Burcu Dogan while working on the Google Drive team
CLI tool to upload object to s3-compatible storage backend and set download policy for it.
typora-s3 CLI tool to upload object to s3-compatible storage backend and set download policy for it. Build $ git clone https://github.com/fengxsong/ty
Sugared logger for console with request id
📝 zlog You don't know what it's like, you don't have a clue If you did you'd find yourselves doing the same thing too Sugared logger for console with
Linux GUI viewer for ZFS pool, dataset and host storage.
This is a WIP in alpha stage. zgui Linux GUI viewer for ZFS pool, dataset and host storage. zgui used libzfs directly and not ZFS command line tools.
Self-hosted web app for encoding files to a target format using distributed computing.
What is Encodarr? Encodarr is a self-hosted web application that encodes video files to a target format using distributed computing to spread the work
GoDrive: A cloud storage system similar to Dropbox or Google Drive, with resilient
Cloud Storage Service Author: Marisa Tania, Ryan Tjakrakartadinata Professor: Matthew Malensek See project spec here: https://www.cs.usfca.edu/~mmalen
A probabilistic data structure service and storage
Skizze ([ˈskɪt͡sə]: german for sketch) is a sketch data store to deal with all problems around counting and sketching using probabilistic data-structu
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values. Authors: Sanjay Ghem
High performance, distributed and low latency publish-subscribe platform.
Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto
The Swift Virtual File System
*** This project is not maintained anymore *** The Swift Virtual File System SVFS is a Virtual File System over Openstack Swift built upon fuse. It is
SeaweedFS a fast distributed storage system for blobs, objects, files, and data lake, for billions of files
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
A distributed key value store in under 1000 lines. Used in production at comma.ai
minikeyvalue Fed up with the complexity of distributed filesystems? minikeyvalue is a ~1000 line distributed key value store, with support for replica
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go Overview Goofys allows you to mount an S3 bucket as a filey system. It's a
A user-space file system for interacting with Google Cloud Storage
gcsfuse is a user-space file system for interacting with Google Cloud Storage. Current status Please treat gcsfuse as beta-quality software. Use it fo
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.
Distributed disk storage database based on Raft and Redis protocol.
IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
Consul Website: https://www.consul.io Tutorials: HashiCorp Learn Forum: Discuss Consul is a distributed, highly available, and data center aware solut
Cloud-Native distributed storage built on and for Kubernetes
Longhorn Build Status Engine: Manager: Instance Manager: Share Manager: Backing Image Manager: UI: Test: Release Status Release Version Type 1.1 1.1.2
Storage Orchestration for Kubernetes
What is Rook? Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for a diverse se
s3git: git for Cloud Storage. Distributed Version Control for Data.
s3git: git for Cloud Storage. Distributed Version Control for Data. Create decentralized and versioned repos that scale infinitely to 100s of millions of files. Clone huge PB-scale repos on your local SSD to make changes, commit and push back. Oh yeah, it dedupes too and offers directory versioning.
Perkeep (née Camlistore) is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content.
Perkeep is your personal storage system. It's a way to store, sync, share, import, model, and back up content. Keep your stuff for life. For more, see
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr
High Performance, Kubernetes Native Object Storage
MinIO Quickstart Guide MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Ama
A simple distributed kv system from scratch
SimpleKV A simple distributed key-value storage system based on bitcask from scratch. Target Here are some basic requirements: LRU Cache. An index sys
Easy to use Raft library to make your app distributed, highly available and fault-tolerant
An easy to use customizable library to make your Go application Distributed, Highly available, Fault Tolerant etc... using Hashicorp's Raft library wh
A simple distributed key-value store by using hashicorp/raft
raftkv This repository holds a simple distributed key-value store by using hashicorp/raft. raftkv provides gRPC and HTTP APIs. Please take a look API
Dynamically provisioning persistent local storage with Kubernetes
Local Path Provisioner Overview Local Path Provisioner provides a way for the Kubernetes users to utilize the local storage in each node. Based on the
Distributed disk storage database based on Raft and Redis protocol.
IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and
Archiving Street Art in a distributed manner
Graffiti Repository provides a tool for graffiti community to keep track and preserving street artwork in their areas. Features Distributed storage Th
The TinyKV course builds a key-value storage system with the Raft consensus algorithm
The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.
[WIP] Cheap, portable and secure NAS based on the Raspberry Pi Zero - with encryption, backups, and more
PortaDisk - Affordable Raspberry Pi Portable & Secure NAS Project Project Status: Early work in progress. web-unlock is still not ready for production
A gateway to expose s3 standard API for any storage type.
s3-gateway this project is used to be compatible with any other storage type. there are tons of object storage services in the internat. However, many
Paxoskv: a Naive and Basic paxos kv storage
paxoskv: a Naive and Basic paxos kv storage 这个repo 目前仅是用于学习的实例代码. 这是一个基于paxos, 只有200行代码的kv存储系统的简单实现, 以最简洁的形式展示paxos如何运行, 作为 可靠分布式系统-paxos的直观解释 这篇教程中的代
A distributed append only commit log used for quick writes and reads to any scale
Maestro-DB A distributed append only commit log used for quick writes and reads to any scale Part 1 - Scaffolding Part-1 Notes Going to start off with
Provide open, community driven reusable components for building distributed applications
Components Contrib The purpose of Components Contrib is to provide open, community driven reusable components for building distributed applications. T
This is a simple file storage server. User can upload file, delete file and list file on the server.
Simple File Storage Server This is a simple file storage server. User can upload file, delete file and list file on the server. If you want to build a
rgkv is a distributed kv storage service using raft consensus algorithm.
rgkv rgkv is a distributed kv storage service using raft consensus algorithm. Get/put/append operation High Availability Sharding Linearizability Tabl
A lightweight and efficient messaging gateway server for distributed game servers, written in Go.
Overview channeld is an open source, light-weight and efficient messaging gateway server designed for distributed game servers (typically MMO) and oth
A key-value storage transaction interpretator.
kv-txn-interpreter A key-value storage transaction interpreter, which provides an etcd-like transaction interface to help you build a transaction over
An implementation of a distributed KV store backed by Raft tolerant of node failures and network partitions 🚣
barge A simple implementation of a consistent, distributed Key:Value store which uses the Raft Concensus Algorithm. This project launches a cluster of
K8s local storage sync for stateful set's using microk8s-hostpath storage classe
Local Storage Sync for microk8s-hostpath The goal is to be able to sync stateful sets between the different nodes of a cluster to allow the data to be
Distributed Systems 2021 -- Miniproject 3
Mini_Project3 == A Distributed Auction System == You must implement a distributed auction system using replication: a distributed component which hand
Network connecter for storage.
database Quick fast connection to database use gorm. Installation $ go get -u github.com/coolstina/connecter Example package main import ( "fmt" "
Distributed Systems 2021 - Miniproject 3
Distributed Systems 2021 -- Miniproject 3 Hand-in Date: 1 December 2021 (at 23:59) What to submit on learnit: a single zip-compressed file containing:
Enable requests served by caddy for distributed tracing via The OpenTracing Project.
caddy-opentracing Enable requests served by caddy for distributed tracing via The OpenTracing Project. Dependencies The Go OpenTracing Library Jaeger,
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.
The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.
cloneMAP: cloud-native Multi-Agent Platform
cloneMAP: cloud-native Multi-Agent Platform cloneMAP is a multi-agent platform (MAP) that is designed to run in a cloud environment based on Kubernete
A federated community framework
Go Distributed A federated community framework Overview Build federated communities using the same API. Distributed provides a single reusable API for
PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage.
中文 | English PaddleDTX PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage. It solves the d
A linearizability distributed database by raft and wisckey.
AlfheimDB A linearizability distributed database by raft and wisckey, which supports redis client. Build This project build by mage, you will need ins
Void is a zero storage cost large file sharing system.
void void is a zero storage cost large file sharing system. License Copyright © 2021 Changkun Ou. All rights reserved. Unauthorized using, copying, mo
The repository aims to share some useful about distributed system
The repository aims to share some useful about distributed system
CSI for S3 compatible SberCloud Object Storage Service
sbercloud-csi-obs CSI for S3 compatible SberCloud Object Storage Service This is a Container Storage Interface (CSI) for S3 (or S3 compatible) storage
CSI driver NVMf mainly supports RDMA and TCP for Software Defined Storage by NVMf
CSI NVMf driver Overview This is a repository for NVMe-oF CSI Driver. Currently it implements bare minimum of th CSI spec. Requirements The CSI NVMf d
Service Discovery and Governance Center for Distributed and Microservice Architecture
Polaris: Service Discovery and Governance English | 简体中文 README: Introduction Components Getting started Chat group Contribution Visit website to lear
PgSQL compatible on distributed database TiDB
TiDB for PostgreSQL Introduction TiDB for PostgreSQL is an open source launched by Digital China Cloud Base to promote and integrate into the open sou
Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload.
Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload. Run tools like masscan, puredns, ffuf, httpx or a
A simple but powerful distributed lock
nlock A simple but powerful distributed lock Get Started Download go get github.com/inuggets/nlock Usage Redis lock import lock "github.com/inuggets/
A lightweight, distributed and reliable message queue based on Redis
nmq A lightweight, distributed and reliable message queue based on Redis Get Started Download go get github.com/inuggets/nmq Usage import "github.com
MIT 6.824: Distributed Systems
MIT 6.824 is a core 12-unit graduate subject with lectures, readings, programming labs, an optional project, a mid-term exam, and a final exam.
OpenAIOS is an incubating open-source distributed OS kernel based on Kubernetes for AI workloads
OpenAIOS is an incubating open-source distributed OS kernel based on Kubernetes for AI workloads. OpenAIOS-Platform is an AI development platform built upon OpenAIOS for enterprises to develop and deploy AI applications for production.
Frisbee is a Kubernetes-native platform for exploring, testing, and benchmarking distributed applications.
Why Frisbee ? Frisbee is a next generation platform designed to unify chaos testing and perfomance benchmarking. We address the key pain points develo
Key-Value Storage written in Go.
kvs kvs is an in-memory key-value storage written in Go. It has 2 different usage. It can be used as a package by importing it to your code or as a se
Store files in Telegram messages for free and access them from a modern Web UI
Telegram Storage Store files in Telegram messages for free and access them in a nice web UI. Telegram allows to store files (of max 2GB each) for a un
A simple RESTful API for storing and retrieving data during MRI image reconstructions
MRD Storage Server The MRD Storage Server provides a simple RESTful API for storing and retrieving data during MRI image reconstructions. It supports:
A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.
A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.
The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.
TinySQL is a course designed to teach you how to implement a distributed relational database in Go
TinySQL TinySQL is a course designed to teach you how to implement a distributed relational database in Go. TinySQL is also the name of the simplifed
Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations
Filestore Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations (store named binar
A distributed logger micro-service built in go
Distributed Logger A distributed logger micro-service built in go using gRPC for internal communication with other micro-services and JSON for externa
Distributed merge sort for large sets across nodes in a network
distMergeSort An implementation of mergesort distributed across nodes used to sort large sets. Introduction Merge sort partitions sets so that they ca
Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.
Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr
Distributed key-value store
Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1
A Redis-compatible server with PostgreSQL storage backend
postgredis A wild idea of having Redis-compatible server with PostgreSQL backend. Getting started As a binary: ./postgredis -addr=:6380 -db=postgres:/
Secret management toolchain
Harp TL;DR. Why harp? Use cases How does it work? Like a Data pipeline but for secret Immutable transformation What can I do? FAQ License Homebrew ins
Distributed, offline-first bug tracker embedded in git, with bridges
Distributed, offline-first bug tracker embedded in git, with bridges