500 Resources
Golang distributed-storage Libraries
Juno 译名朱诺。这个名字来源于古罗马神话中的众神之母。它是斗鱼的微服务管理系统, 如同众神之母一样守护着所有微服务的系统。
JUNO - A distributed application management system Introduction JUNO(朱诺) 是斗鱼数据服务组研发的分布式服务管理系统,核心功能为配置中心,能集中化管理应用不同环境、不同机房的配置;并且集成了监控中心、测试平台、日志中心等功能模块,
open source training courses about distributed database and distributed systemes
Welcome to learn Talent Plan Courses! Talent Plan is an open source training program initiated by PingCAP. It aims to create or combine some open sour
Distributed WebSocket broker
dSock dSock is a distributed WebSocket broker (in Go, using Redis). Clients can authenticate & connect, and you can send text/binary message as an API
Distributed RTC System by pure Go and Flutter
ION is a distributed real-time communication system, the goal is to chat anydevice, anytime, anywhere! Distributed Real-time Communication System ION
Implementation of the NELI leader election protocol for Go and Kafka
goNELI Implementation of the NELI leader election protocol for Go and Kafka. goNELI encapsulates the 'fast' variation of the protocol, running in excl
Lockgate is a cross-platform locking library for Go with distributed locks using Kubernetes or lockgate HTTP lock server as well as the OS file locks support.
Lockgate Lockgate is a locking library for Go. Classical interface: 2 types of locks: shared and exclusive; 2 modes of locking: blocking and non-block
A distributed Configuration Center server that manages config in a container. The container is composed of fields (abstract layer includes: KV, LIST, DICT type). The Field contains basic datatypes (int, float, bool, string, list, dict).
cassem config assembler from key-value pairs' container which include basic datatypes, such as int, string, float, bool, list, dict Features HTTP Rest
Database migrations. CLI and Golang library.
Database migrations written in Go. Use as CLI or import as library.
a distributed chat system which can be used as chat rooms or state synchronization
unitalk is a distributed chat system which can be used as chat rooms or state synchronization. unitalk registers itself on zookeeper when it start up, use redis cluster to broadcast messages or state, and permanent messages or state to kafka and then save to db.
CRAQ (Chain Replication with Apportioned Queries) in Go
CRAQ is a replication protocol that allows reads from any replica while still maintaining strong consistency. CRAQ should provide better read throughput than Raft and Paxos. Read performance grows linearly with the number of nodes added to the system. Network chatter is significantly lower compared to Raft and Paxos.
TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage.
TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage. By leveraging smart contracts, client-side e
A distributed, proof of stake blockchain designed for the financial services industry.
Provenance Blockchain Provenance is a distributed, proof of stake blockchain designed for the financial services industry.
Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.
Dapr is a portable, serverless, event-driven runtime that makes it easy for developers to build resilient, stateless and stateful microservices that run on the cloud and edge and embraces the diversity of languages and developer frameworks.
TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage
TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage. By leveraging smart contracts, client-side encryption, and sophisticated redundancy (via Reed-Solomon codes), TurtleDex allows users to safely store their data with hosts that they do not know or trust.
distributed monitoring system
OWL OWL 是由国内领先的第三方数据智能服务商 TalkingData 开源的一款企业级分布式监控告警系统,目前由 Tech Operation Team 持续开发更新维护。 OWL 后台组件全部使用 Go 语言开发,Go 语言是 Google 开发的一种静态强类型、编译型、并发型,并具有垃圾回
Distributed, lock-free, self-hosted health checks and status pages
Checkup is distributed, lock-free, self-hosted health checks and status pages, written in Go. It features an elegant, minimalistic CLI and an idiomati
Micro is a platform for cloud native development
Micro Overview Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provid
CNCF Jaeger, a Distributed Tracing Platform
Jaeger - a Distributed Tracing System Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing platform created by Uber Technologies and do
moss - a simple, fast, ordered, persistable, key-val storage library for golang
moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library.
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.
Distributed, fault-tolerant key-value storage written in go.
A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.
Go Open Source, Distributed, Simple and efficient Search Engine
Go Open Source, Distributed, Simple and efficient full text search engine.
A distributed unique ID generator inspired by Twitter's Snowflake
Sonyflake is a distributed unique ID generator inspired by Twitter's Snowflake.
a high-performance, POSIX-ish Amazon S3 file system written in Go
Goofys allows you to mount an S3 bucket as a filey system.
Web-based Cloud Gaming service for Retro Game
CloudRetro provides an open-source cloud gaming platform for retro games. It started as an experiment for testing cloud gaming performance with WebRTC and libretro, and now it aims to deliver the most modern and convenient gaming experience through the technology.
NewSQL distributed storage database based on micro service framework
QLite 是基于微服务的 NewSQL 型数据库系统,与传统的一体化数据库不同,该系统将本该内置的多种数据结构(STL)拆分成多个服务模块,每个模块都是独立的一个节点,每个节点都与其主网关进行连接,从而形成分布式存储结构。
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is an open-source POSIX file system built on top of Redis and object storage (e.g. Amazon S3), designed and optimized for cloud native environ
Collyzar - A distributed redis-based framework for colly.
Collyzar A distributed redis-based framework for colly. Collyzar provides a very simple configuration and tools to implement distributed crawling/scra
An encrypted object storage system with unlimited space backed by Telegram.
TGStore An encrypted object storage system with unlimited space backed by Telegram. Please only upload what you really need to upload, don't abuse any
CSI Driver for dynamic provisioning of Persistent Local Volumes for Kubernetes using LVM.
OpenEBS LVM CSI Driver CSI driver for provisioning Local PVs backed by LVM and more. Project Status Currently the LVM CSI Driver is in alpha
⟁ Tendermint Core (BFT Consensus) in Go
Tendermint Byzantine-Fault Tolerant State Machines. Or Blockchain, for short. Branch Tests Coverage Linting master Tendermint Core is Byzantine Fault
Parallel S3 and local filesystem execution tool.
s5cmd Overview s5cmd is a very fast S3 and local filesystem execution tool. It comes with support for a multitude of operations including tab completi
Google Cloud Client Libraries for Go.
Google Cloud Client Libraries for Go Go packages for Google Cloud Platform services. import "cloud.google.com/go" To install the packages on your syst
[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。
Goribot 一个分布式友好的轻量的 Golang 爬虫框架。 完整文档 | Document !! Warning !! Goribot 已经被迁移到 Gospider|github.com/zhshch2002/gospider。修复了一些调度问题并分离了网络请求部分到另一个仓库。此仓库会继续
Fully featured and highly configurable SFTP server with optional FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob
SFTPGo Fully featured and highly configurable SFTP server with optional FTP/S and WebDAV support, written in Go. Several storage backends are supporte
High Performance, Kubernetes Native Object Storage
MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag
Distributed reliable key-value store for the most critical data of a distributed system
etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.
Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An
Secure software enclave for storage of sensitive information in memory.
MemGuard Software enclave for storage of sensitive information in memory. This package attempts to reduce the likelihood of sensitive data being expos
Instant Kubernetes-Native Application Observability
What is Pixie? Pixie gives you instant visibility by giving access to metrics, events, traces and logs without changing code. We're building up Pixie
CNCF Jaeger, a Distributed Tracing Platform
Jaeger - a Distributed Tracing System Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing platform created by Uber Technologies and do
An enhanced HTTP client for Go
Heimdall Description Installation Usage Making a simple GET request Creating a hystrix-like circuit breaker Creating a hystrix-like circuit breaker wi
A distributed unique ID generator of using Sonyflake and encoded by Base58
Indigo About A distributed unique ID generator of using Sonyflake and encoded by Base58. ID max length is 11 characters by unsigned int64 max value. A
Go Micro is a framework for distributed systems development
Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop
Machinery is an asynchronous task queue/job queue based on distributed message passing.
Machinery Machinery is an asynchronous task queue/job queue based on distributed message passing. V2 Experiment First Steps Configuration Lock Broker
Asynq: simple, reliable, and efficient distributed task queue in Go
Asynq Overview Asynq is a Go library for queueing tasks and processing them asynchronously with workers. It's backed by Redis and is designed to be sc
Bayesian text classifier with flexible tokenizers and storage backends for Go
Shield is a bayesian text classifier with flexible tokenizer and backend store support Currently implemented: Redis backend English tokenizer Example
Distributed hyperparameter optimization framework, inspired by Optuna.
Goptuna Distributed hyperparameter optimization framework, inspired by Optuna [1]. This library is particularly designed for machine learning, but eve
A consistent distributed data store.
Doozer What Is It? Doozer is a highly-available, completely consistent store for small amounts of extremely important data. When the data changes, it
Go language interface to the Libcircle distributed-queue API
Circle Description The Circle package provides a Go interface to the Libcircle distributed-queue API. Despite the name, Circle has nothing to do with
A cloud native distributed streaming network telemetry.
Panoptes Streaming Panoptes Streaming is a cloud native distributed streaming network telemetry. It can be installed as a single binary or clustered n
Industrial IoT Messaging and Device Management Platform
Mainflux Mainflux is modern, scalable, secure, open-source, and patent-free IoT cloud platform written in Go. It accepts user and thing (sensor, actua
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
glow Purpose Glow is providing a library to easily compute in parallel threads or distributed to clusters of machines. This is written in pure Go. I a
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.
Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built
Storage and image processing server written in Go
Mort An S3-compatible image processing server written in Go. Still in active development. Features HTTP server Resize, Rotate, SmartCrop Convert (JPEG
darkroom - An image proxy with changeable storage backends and image processing engines with focus on speed and resiliency.
Darkroom - Yet Another Image Proxy Introduction Darkroom combines the storage backend and the image processor and acts as an Image Proxy on your image
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.
go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice. A simple in-memory database and table implementation are provided, and you can query any data source you want by implementing a few interfaces.
Lightweight, facility, high performance golang based game server framework
Nano Nano is an easy to use, fast, lightweight game server networking library for Go. It provides a core network architecture and a series of tools an
Scalable Distributed Game Server Engine with Hot Swapping in Golang
GoWorld Scalable Distributed Game Server Engine with Hot Reload in Golang Features Architecture Introduction Get GoWorld Manage GoWorld Servers Demos
Abstract File Storage
afs - abstract file storage Please refer to CHANGELOG.md if you encounter breaking changes. Motivation Introduction Usage Matchers Content modifiers S
Skynet is a framework for distributed services in Go.
##Introduction Skynet is a communication protocol for building massively distributed apps in Go. It is not constrained to Go, so it will lend itself n
*DEPRECATED* Please use https://gopkg.in/redsync.v1 (https://github.com/go-redsync/redsync)
Redsync.go This package is being replaced with https://gopkg.in/redsync.v1. I will continue to maintain this package for a while so that its users do
The pure golang implementation of nanomsg (version 1, frozen)
mangos NOTE: This is the legacy version of mangos (v1). Users are encouraged to use mangos v2 instead if possible. No further development is taking pl
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.
What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are
A distributed lock service in Go using etcd
locker A distributed lock service client for etcd. What? Why? A distributed lock service is somewhat self-explanatory. Locking (mutexes) as a service
⟁ Tendermint Core (BFT Consensus) in Go
Tendermint Byzantine-Fault Tolerant State Machines. Or Blockchain, for short. Branch Tests Coverage Linting master Tendermint Core is Byzantine Fault
A Go library for master-less peer-to-peer autodiscovery and RPC between HTTP services
sleuth sleuth is a Go library that provides master-less peer-to-peer autodiscovery and RPC between HTTP services that reside on the same network. It w
Simplified distributed locking implementation using Redis
redislock Simplified distributed locking implementation using Redis. For more information, please see examples. Examples import ( "fmt" "time"
A library that implements the outboxer pattern in go
Outboxer Outboxer is a go library that implements the outbox pattern. Getting Started Outboxer was designed to simplify the tough work of orchestratin
High-Performance server for NATS, the cloud native messaging system.
NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Fo
Micro is a platform for cloud native development
Micro Overview Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provid
Lightweight, fault-tolerant message streams.
Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extend
MySQL Backed Locking Primitive
go-mysql-lock go-mysql-lock provides locking primitive based on MySQL's GET_LOCK Lock names are strings and MySQL enforces a maximum length on lock na
Go Micro is a framework for distributed systems development
Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
glow Purpose Glow is providing a library to easily compute in parallel threads or distributed to clusters of machines. This is written in pure Go. I a
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.
Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built
Flowgraph package for scalable asynchronous system development
flowgraph Getting Started go get -u github.com/vectaport/flowgraph go test Links Wiki Slides from Minneapolis Golang Meetup, May 22nd 2019 Overview F
High performance, distributed and low latency publish-subscribe platform.
Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto
A feature complete and high performance multi-group Raft library in Go.
Dragonboat - A Multi-Group Raft library in Go / 中文版 News 2021-01-20 Dragonboat v3.3 has been released, please check CHANGELOG for all changes. 2020-03
distributed data sync with operational transformation/transforms
DOT The DOT project is a blend of operational transformation, CmRDT, persistent/immutable datastructures and reactive stream processing. This is an im
cmd tool for automatic storage and comparison of benchmarks results
prettybenchcmp prettybenchcmp is cmd tool for storage and comparison of benchmarks results. There is a standard tool benchcmp, but I don't think that
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.
Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With
Distributed reliable key-value store for the most critical data of a distributed system
etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order
Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)
gokv Simple key-value store abstraction and implementations for Go Contents Features Simple interface Implementations Value types Marshal formats Road
Database migrations. CLI and Golang library.
migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to
VictoriaMetrics: fast, cost-effective monitoring solution and time series database
VictoriaMetrics VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. It is available in binary release
A distributed key-value store. On Disk. Able to grow or shrink without service interruption.
Vasto A distributed high-performance key-value store. On Disk. Eventual consistent. HA. Able to grow or shrink without service interruption. Vasto sca
TiDB is an open source distributed HTAP database compatible with the MySQL protocol
Slack Channel Twitter: @PingCAP Reddit Mailing list: lists.tidb.io For support, please contact PingCAP What is TiDB? TiDB ("Ti" stands for Titanium) i
The lightweight, distributed relational database built on SQLite.
rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it grace
moss - a simple, fast, ordered, persistable, key-val storage library for golang
moss moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library. moss stands for "memory-oriented s
Native GraphQL Database with graph backend
The Only Native GraphQL Database With A Graph Backend. Dgraph is a horizontally scalable and distributed GraphQL database with a graph backend. It pro
Secure storage for personal records built to comply with GDPR
Databunker Databunker is a Personally Identifiable Information (PII) Data Storage Service built to Comply with GDPR and CCPA Privacy Requirements. Pro
CockroachDB - the open source, cloud-native distributed SQL database.
CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. What is CockroachDB? Docs Quickstart C
Eventually consistent distributed in-memory cache Go library
bcache A Go Library to create distributed in-memory cache inside your app. Features LRU cache with configurable maximum keys Eventual Consistency sync
Fast in-memory key:value store/cache with TTL
MCache library go-mcache - this is a fast key:value storage. Its major advantage is that, being essentially a thread-safe . map[string]interface{} wit
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.
Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An
Go language interface to Swift / Openstack Object Storage / Rackspace cloud files (golang)
Swift This package provides an easy to use library for interfacing with Swift / Openstack Object Storage / Rackspace cloud files from the Go Language
Next generation distributed, event-driven, parallel config management!
mgmt: next generation config management! About: Mgmt is a real-time automation tool. It is familiar to existing configuration management software, but
Composable, observable and performant config handling for Go for the distributed processing era
Konfig Composable, observable and performant config handling for Go. Written for larger distributed systems where you may have plenty of configuration
persistent storage for flags in go
ingo is a simple Go library helping you to persist flags in a ini-like config file. Features and limitations Requires Go 1.5 or later automatically cr