500 Resources
Golang distributed-storage Libraries
Distributed Lab 3: Message Broker in Go
Distributed Lab 3: Message Broker in Go Using the lab sheet There are two ways to use the lab sheet, you can either: create a new repo from this templ
decentralized,distributed,peer-to-peer database.
P2PDB 中文 | English 简介 P2PDB(p2p数据库),是一个去中心化、分布式、点对点数据库、P2PDB使用IPFS为其数据存储和IPFS Pubsub自动与对等方同步数据。P2PDB期望打造一个去中心化的分布式数据库,使P2PDB 成为去中心化应用程序 (dApps)、区块链应用程
A distributed Layer 2 Direct Server Return (L2DSR) load balancer for Linux using XDP/eBPF
VC5 A distributed Layer 2 Direct Server Return (L2DSR) load balancer for Linux using XDP/eBPF This is very much a proof of concept at this stage - mos
MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads.
What is MatrixOne? MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads. It provides an end-to-end data
An implementation of the Filecoin Distributed Storage Network
Project Lotus - 莲 Lotus is an implementation of the Filecoin Distributed Storage Network. For more details about Filecoin, check out the Filecoin Spec
Cache list, count with filter param golang, using struct, hashkey
Dumbcache Cache list, count with filter param golang, using struct, hashkey Structure we hash your request object to md5 hashing and add a prefix coun
🦉owlcache is a lightweight, high-performance, non-centralized, distributed Key/Value memory-cached data sharing application written by Go
🦉owlcache is a lightweight, high-performance, non-centralized, distributed Key/Value memory-cached data sharing application written by Go . keyword : golang cache、go cache、golang nosql
⚙️ Dead Simple Config Management, load and persist config without having to think about where and how.
Configo Dead Simple Config Management, load and persist config without having to think about where and how. Install go get github.com/UltiRequiem/conf
Distributed Lab 2: RPC in Go
Distributed Lab 2: RPC in Go Using the lab sheet There are two ways to use the lab sheet, you can either: create a new repo from this template - this
Coroutine based Deterministic 2PC Distributed Transaction
Enviorment set up off the mod go env -w GO111MODULE=off change to GPATH to project directory go env -w GOPATH=$HOME/coroprove Usage This project serve
A set of components that can be composed into a highly available metric system with unlimited storage capacity
Overview Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added
🌐 (Web 3.0) Pastebin built on IPFS, securely served by Distributed Web and Edge Network.
pastebin-ipfs 简体中文 (IPFS Archivists) Still in development, Pull Requests are welcomed. Pastebin built on IPFS, securely served by Distributed Web and
Secure Distributed Thanos Deployment using an Observability Cluster
Atlas Status: BETA - I don't expect breaking changes, but still possible. Atlas, forced by Zeus to support the heavens and the skies on his shoulders.
Stream data into Google BigQuery concurrently using InsertAll() or BQ Storage.
bqwriter A Go package to write data into Google BigQuery concurrently with a high throughput. By default the InsertAll() API is used (REST API under t
Distributed Lab 2: RPC in Go
Distributed Lab 2: RPC in Go Using the lab sheet There are two ways to use the lab sheet, you can either: create a new repo from this template - this
An open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developersAn open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developers
Developer-oriented Continuous Delivery Product English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? How to
implementation of some distributed system techniques
Distributed Systems These applications were built with the objective of studding a distributed systems using the most recent technics. The main ideia
TinyHat.Me: Microservices deployed with Kubernetes that enable users to propose hat pictures and try on hats from a user-curated database.
Click here to see the "buggy" version 🐛 The Scenario TinyHat.Me is an up and coming startup that provides an API to allow users to try on tiny hats v
Container Storage Interface driver for Synology NAS
Synology CSI Driver for Kubernetes The official Container Storage Interface driver for Synology NAS. Container Images & Kubernetes Compatibility Drive
Sync distributed sets using bloom filters
goSetReconciliation An implementation to sync distributed sets using bloom filters. Based on the paper "Low complexity set reconciliation using Bloom
entcache - An experimental cache driver for ent with variety of storage options
entcache An experimental cache driver for ent with variety of storage options, such as: A context.Context-based cache. Usually, attached to an HTTP re
Azure Data Lake Storage Account Share 9p File System
dlfs Azure storage account (data lake) v2 as a 9p file system. Fork of abfs. Written in Go. Created during the 2021 MS hackathon. Build Currently the
Go package for running Linux distributed shell commands via SSH.
Go package for running Linux distributed shell commands via SSH.
Simple Golang API to demonstrate file upload to fireabase storage and retrieving url of uploaded file.
go-firebase-storage -Work in progress 🛠️ Simple Golang API that uses Firebase as its backend to demonstrate various firebase services using Go such a
Eunomia is a distributed application framework that support Gossip protocol, QuorumNWR algorithm, PBFT algorithm, PoW algorithm, and ZAB protocol and so on.
Introduction Eunomia is a distributed application framework that facilitates developers to quickly develop distributed applications and supports distr
Versioned model registry suitable for temporary in-training storage and permanent storage
Cogment Model Registry Cogment is an innovative open source AI platform designed to leverage the advent of AI to benefit humankind through human-AI co
A distributed locking library built on top of Cloud Spanner and TrueTime.
A distributed locking library built on top of Cloud Spanner and TrueTime.
Rook is an open source cloud-native storage orchestrator for Kubernetes
Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments.
A distributed systems library for Kubernetes deployments built on top of spindle and Cloud Spanner.
hedge A library built on top of spindle and Cloud Spanner that provides rudimentary distributed computing facilities to Kubernetes deployments. Featur
A Go client implementing a client-side distributed consumer group client for Amazon Kinesis.
Kinesumer is a Go client implementing a client-side distributed consumer group client for Amazon Kinesis.
SpiceDB is a Zanzibar-inspired database that stores, computes, and validates application permissions.
SpiceDB is a Zanzibar-inspired database that stores, computes, and validates application permissions. Developers create a schema that models t
A Go client and CLI for Filecoin Storage Auctions.
go-auctions-client A Go library and CLI to interact with Filecoin Storage Auctions. Join us on our public Slack channel for news, discussions, and sta
go-test-trace is like go test but it also generates distributed traces.
go-test-trace go-test-trace is like go test but it also generates distributed traces. Generated traces are exported in OTLP to a OpenTelemetry collect
Multithreaded key value pair store using thread safe locking mechanism allowing concurrent reads
Project Amnesia A Multi-threaded key-value pair store using thread safe locking mechanism allowing concurrent reads. Curious to Try it out?? Check out
painless task queue manager for shell commands with an intuitive cli interface (execute shell commands in distributed cloud-native queue manager).
EXEQ DOCS STILL IN PROGRESS. Execute shell commands in queues via cli or http interface. Features Simple intuitive tiny cli app. Modular queue backend
SigNoz helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
Monitor your applications and troubleshoot problems in your deployed applications, an open-source alternative to DataDog, New Relic, etc. Documentatio
Terraform provider for the Minio object storage.
terraform-provider-minio A Terraform provider for Minio, a self-hosted object storage server that is compatible with S3. Check out the documenation on
A collection of authentication Go packages related to OIDC, JWKs and Distributed Claims.
cap (collection of authentication packages) provides a collection of related packages which enable support for OIDC, JWT Verification and Distributed Claims.
SFTPGo - Fully featured and highly configurable SFTP server with optional FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob
SFTPGo - Fully featured and highly configurable SFTP server with optional FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob
StaticBackend is a simple backend server API handling user mgmt, database, storage and real-time component
StaticBackend is a simple backend that handles user management, database, file storage, forms, and real-time experiences via channel/topic-based communication for web and mobile applications.
Container Storage Interface components for SPIFFE
SPIFFE CSI Driver WARNING: This project is in the "Development" phase of the SPIFFE Project Maturity Phases. A Container Storage Interface driver for
MySQL Storage engine conversion,Support mutual conversion between MyISAM and InnoDB engines.
econvert MySQL Storage engine conversion 简介 此工具用于MySQL存储引擎转换,支持CTAS和ALTER两种模式,目前只支持MyISAM和InnoDB存储引擎相互转换,其它引擎尚不支持。 注意:当对表进行引擎转换时,建议业务停止访问或者极少量访问时进行。 原
A distributed fault-tolerant order book matching engine
Go - Between A distributed fault-tolerant order book matching engine. Features Limit orders Market orders Order book depth Calculate market price for
Golang implementation of distributed mutex on Azure lease blobs
Distributed Mutex on Azure Lease Blobs This package implements distributed lock available for multiple processes. Possible use-cases include exclusive
Asynchronous data replication for Kubernetes volumes
VolSync VolSync asynchronously replicates Kubernetes persistent volumes between clusters using either rsync or rclone. It also supports creating backu
Carina: an high performance and ops-free local storage for kubernetes
Carina English | 中文 Background Storage systems are complex! There are more and more kubernetes native storage systems nowadays and stateful applicatio
storage interface for local disk or AWS S3 (or Minio) platform
storage interface for local disk or AWS S3 (or Minio) platform
Example patterns for distributed service
Cloud Native Tulisan ini akan mengulas hasil pembelajaran dari beberapa sumber #learn-from-books. Dan terinspirasi dari obrolan The Pursuit of Product
a go client for distributed transaction manager DTM. go语言分布式事务管理器DTM的客户端SDK
a client for distributed transaction manager dtm dtmcli 是分布式事务管理器dtm的客户端sdk dtmcli 这个库的代码与dtm下的dtmcli代码保持完全同步,如果您需要线上使用,那么当前的dtmcli相关的依赖非常少,对于最终打包的镜像也
Abusing Discord for unlimited cloud storage
Discord Cloud Storage Abusing Discord's servers for unlimited cloud storage! So, what is this? Infamous 8MB limit for non-nitro users can get pretty a
gathering distributed key-value datastores to become a cluster
go-ds-cluster gathering distributed key-value datastores to become a cluster About The Project This project is going to implement go-datastore in a fo
This library contains utilities that are useful for building distributed services.
Grafana Dskit This library contains utilities that are useful for building distributed services. Current state This library is still in development. D
Open-Local is a local disk management system composed of multiple components.
Open-Local is a local disk management system composed of multiple components. With Open-Local, using local storage in Kubernetes will be as simple as centralized storage.
IceFireDB - Distributed disk storage system based on Raft and RESP protocol.
Distributed disk storage database based on Raft and Redis protocol.
Distributed cache and in-memory key/value data store.
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.
A distributed key value store in under 1000 lines. Used in production at comma.ai
minikeyvalue Fed up with the complexity of distributed filesystems? minikeyvalue is a ~1000 line distributed key value store, with support for replica
Tarmac is a unique framework designed for the next generation of distributed systems
Framework for building distributed services with Web Assembly
A magic shim for Docker credential helpers
docker-credential-magic Overview Installation Usage How to use docker-credential-magic Local setup How to use docker-credential-magician Including a s
Jaeger ClickHouse storage plugin implementation
This is implementation of Jaeger's storage plugin for ClickHouse.
gdcache is a pure non-intrusive distributed cache library implemented by golang
gdcache is a pure non-intrusive distributed cache library implemented by golang, you can use it to implement your own distributed cache
Ananas is an experimental project for kubernetes CSI (Container Storage Interface) by using azure disk. Likewise, Ananas is the name of my cute british shorthair.
ananas Ananas is an experimental project for kubernetes CSI (Container Storage Interface) by using azure disk. Likewise, Ananas is the name of my cute
An high performance and ops-free local storage solution for Kubernetes.
Carina carina 是一个CSI插件,在Kubernetes集群中提供本地存储持久卷 项目状态:开发测试中 CSI Version: 1.3.0 Carina architecture 支持的环境 Kubernetes:1.20 1.19 1.18 Node OS:Linux Filesys
goInterLock is golang job/task scheduler with distributed locking mechanism (by Using Redis🔒).
goInterLock is golang job/task scheduler with distributed locking mechanism. In distributed system locking is preventing task been executed in every instant that has the scheduler,
Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload.
Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload. Run tools like masscan, puredns, ffuf, httpx or anything you need and get results quickly!
Rink is a "distributed sticky ranked ring" using etcd.
Rink is a "distributed sticky ranked ring" using etcd. A rink provides role scheduling across distributed processes, with each role only assigned
cloud-native local storage management system
Open-Local是由多个组件构成的本地磁盘管理系统,目标是解决当前 Kubernetes 本地存储能力缺失问题。通过Open-Local,使用本地存储会像集中式存储一样简单。
Redwood is a highly-configurable, distributed, realtime database that manages a state tree shared among many peers
Redwood is a highly-configurable, distributed, realtime database that manages a state tree shared among many peers. Imagine something like a Redux store, but distributed across all users of an application, that offers offline editing and is resilient to poor connectivity.
Jaeger ClickHouse storage plugin implementation
Jaeger ClickHouse Jaeger ClickHouse gRPC storage plugin. This is WIP and it is based on https://github.com/bobrik/jaeger/tree/ivan/clickhouse/plugin/s
Framework for building distributed services with Web Assembly
Tarmac Framework for building distributed services with Web Assembly Tarmac is a unique framework designed for the next generation of distributed syst
Fast time-series data storage server accessible over gRPC
tstorage-server Persistent fast time-series data storage server accessible over gRPC. tstorage-server is lightweight local on-disk storage engine serv
Application written in Go which polls Time-series data at specific intervals and saves to persistent storage
TPoller Server Overview The purpose of this application is to poll time-series data per time interval from any (telemetry) application running a gRPC
a lightweight distributed transaction management service, support xa tcc saga
a lightweight distributed transaction management service, support xa tcc saga
LiveKit - Open source, distributed video/audio rooms over WebRTC
LiveKit is an open source project that provides scalable, multi-user conferencing over WebRTC. It's designed to give you everything you need to build real time video/audio capabilities in your applications.
Secure software enclave for storage of sensitive information in memory.
MemGuard Software enclave for storage of sensitive information in memory. This package attempts to reduce the likelihood of sensitive data being expos
YTask is an asynchronous task queue for handling distributed jobs in golang
YTask is an asynchronous task queue for handling distributed jobs in golang
A Filecoin Network sidecar for miners to bid in storage deal auctions.
bidbot Bidbot is a Filecoin Network sidecar for miners to bid in storage deal auctions. Join us on our public Slack channel for news, discussions, and
Golimit is Uber ringpop based distributed and decentralized rate limiter
Golimit A Distributed Rate limiter Golimit is Uber ringpop based distributed and decentralized rate limiter. It is horizontally scalable and is based
An actor framework for Go
gosiris is an actor framework for Golang. Features Manage a hierarchy of actors (each actor has its own: state, behavior, mailbox, child actors) Deplo
a Framework for creating microservices using technologies and design patterns of Erlang/OTP in Golang
Technologies and design patterns of Erlang/OTP have been proven over the years. Now in Golang. Up to x5 times faster than original Erlang/OTP in terms
Go Micro is a standalone framework for distributed systems development
Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native
SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.
SeaweedFS Sponsor SeaweedFS via Patreon SeaweedFS is an independent Apache-licensed open source project with its ongoing development made possible ent
Distributed Named Pipes
dnpipes Distributed Named Pipes (or: dnpipes) are essentially a distributed version of Unix named pipes comparable to, for example, SQS in AWS or the
Sandglass is a distributed, horizontally scalable, persistent, time sorted message queue.
Sandglass is a distributed, horizontally scalable, persistent, time ordered message queue. It was developed to support asynchronous tasks and message
A realtime distributed messaging platform
Source: https://github.com/nsqio/nsq Issues: https://github.com/nsqio/nsq/issues Mailing List: [email protected] IRC: #nsq on freenode Docs:
Dkron - Distributed, fault tolerant job scheduling system https://dkron.io
Dkron - Distributed, fault tolerant job scheduling system for cloud native environments Website: http://dkron.io/ Dkron is a distributed cron service,
Distributed reliable key-value store for the most critical data of a distributed system
etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel
A distributed key-value storage system developed by Alibaba Group
Product Overview Tair is fast-access memory (MDB)/persistent (LDB) storage service. Using a high-performance and high-availability distributed cluster
CockroachDB - the open source, cloud-native distributed SQL database.
CockroachDB is a cloud-native distributed SQL database designed to build, scale, and manage modern, data-intensive applications. What is CockroachDB?
The lightweight, distributed relational database built on SQLite
rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it grace
Elastic Key-Value Storage With Strong Consistency and Reliability
What is Elasticell? Elasticell is a distributed NoSQL database with strong consistency and reliability. Compatible with Redis protocol Use Elasticell
A distributed MySQL binlog storage system built on Raft
What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl
A distributed and coördination-free log management system
OK Log is archived I hoped to find the opportunity to continue developing OK Log after the spike of its creation. Unfortunately, despite effort, no su
Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)
Jocko Kafka/distributed commit log service in Go. Goals of this project: Implement Kafka in Go Protocol compatible with Kafka so Kafka clients and ser
OpenDILab RL Object Store
Introduction Decision AI Store Installation Prerequisites Linux Python = 3.6 pip install . Quick Start Start Etcd Server di_store etcd_server ./conf/
Verifiable credential system on Cosmos with IBC for Distributed Identities
CertX This is a project designed to demonstrate the use of IBC between different zones in the Cosmos ecosystem for privacy preserving credential manag
golang实现的分布式唯一ID生成器distributed id generator,有全局趋势递增、严防时钟漂移、高可用、高性能等特点
ekko-idgenerator是什么 顾名思义,ekko是一个分布式唯一ID生成器,参考了snowFlake思想,但是并不局限于其设计。 名称由来 英雄联盟的时间刺客ekko 特点 易用,最大限度保证系统的易用性,支持Get与MultiGet; 高并发,单机每秒100w个唯一ID生成; 高可用,理
GoBigdis is a persistent database that implements the Redis server protocol. Any Redis client can interface with it and start to use it right away.
GoBigdis GoBigdis is a persistent database that implements the Redis server protocol. Any Redis client can interface with it and start to use it right
Mahi is an all-in-one HTTP service for file uploading, processing, serving, and storage.
Mahi is an all-in-one HTTP service for file uploading, processing, serving, and storage. Mahi supports chunked, resumable, and concurrent uploads. Mahi uses Libvips behind the scenes making it extremely fast and memory efficient.