500 Resources
Golang distributed-storage Libraries
short-url distributed and high-performance
durl 是一个分布式的高性能短链服务,逻辑简单,并提供了相关api接口,开发人员可以快速接入,也可以作为go初学者练手项目.
a dynamic configuration framework used in distributed system
go-archaius This is a light weight configuration management framework which helps to manage configurations in distributed system The main objective of
Fault tolerant, sharded key value storage written in GoLang
Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult
Provides some convenient API, includes Goid(), AllGoid(), and LocalStorage, which is a goroutine's local storage, just like ThreadLocal in other languages.
routine 中文版 routine encapsulates and provides some easy-to-use, high-performance goroutine context access interfaces, which can help you access corout
High-performance, columnar, in-memory store with bitmap indexing in Go
This package contains a high-performance, columnar, in-memory storage engine that supports fast querying, update and iteration with zero-allocations and bitmap indexing.
Layotto is an application runtime developed using Golang, which provides various distributed capabilities for applications
Layotto is an application runtime developed using Golang, which provides various distributed capabilities for applications, such as state management, configuration management, and event pub/sub capabilities to simplify application development.
An implementation of a distributed access-control server that is based on Google Zanzibar
An implementation of a distributed access-control server that is based on Google Zanzibar - "Google's Consistent, Global Authorization System".
Managing your Kubernetes clusters (including public, private, edge, etc) as easily as visiting the Internet
Clusternet Managing Your Clusters (including public, private, hybrid, edge, etc) as easily as Visiting the Internet. Clusternet (Cluster Internet) is
The Fabric Token SDK is a set of API and services that lets developers create token-based distributed application on Hyperledger Fabric.
The Fabric Token SDK is a set of API and services that let developers create token-based distributed application on Hyperledger Fabric.
DTM 是一款跨语言的分布式事务管理方案,在各类微服务架构中,提供高性能和简单易用的分布式事务服务
轻量级分布式事务管理服务 DTM 是一款跨语言的分布式事务管理方案,在各类微服务架构中,提供高性能和简单易用的分布式事务服务。 特色 跨语言 语言无关,任何语言实现了http方式的服务,都可以接入DTM,用来管理分布式事务。支持go、python、php、nodejs、ruby 多种分布式事务协议支
A Lightweight VPN Built on top of Libp2p for Truly Distributed Networks.
Hyprspace A Lightweight VPN Built on top of Libp2p for Truly Distributed Networks. demo.mp4 Table of Contents A Bit of Backstory Use Cases A Digital N
The Fabric Smart Client is a new Fabric Client that lets you focus on the business processes and simplifies the development of Fabric-based distributed application.
Fabric Smart Client The Fabric Smart Client (FSC, for short) is a new Fabric client-side component whose objective is twofold. FSC aims to simplify th
Store data on Bitcoin for 350 sats/KB up to 185 KB by using P2SH-P2WSH witness scripts
Bitcandle Store data on Bitcoin for 350 sats/KB up to 185 kB by using P2SH-P2WSH witness scripts. 225ed8bc432d37cf434f80717286fd5671f676f12b573294db72
simple cloud made in GO
simple cloud made in GO im doing this for fun and practice its a really simple cloud and its finished but the frontend its a little bit shitty.If you
Ent adapter for Casbin
Ent-Adapter Ent Adapter is the ent adapter for Casbin. With this library, Casbin can load policy from PostgresSQL/Mysql or save policy to it. Installa
tstorage is a lightweight local on-disk storage engine for time-series data
tstorage is a lightweight local on-disk storage engine for time-series data with a straightforward API. Especially ingestion is massively opt
topolvm operator provide kubernetes local storage which is light weight and high performance
Topolvm-Operator Topolvm-Operator is an open source cloud-native local storage orchestrator for Kubernetes, which bases on topolvm. Supported environm
LFU Redis implements LFU Cache algorithm using Redis as data storage
LFU Redis cache library for Golang LFU Redis implements LFU Cache algorithm using Redis as data storage LFU Redis Package gives you control over Cache
Storj is building a decentralized cloud storage network
Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.
GTA(Go Task Async) is a lightweight reliable asynchronous task and transaction message library for Golang
GTA (Go Task Async) is a lightweight and reliable asynchronous task and transaction message library for by golang.
OCI Registry As Storage
OCI Registry As Storage 🚧 This project is currently under active development. The API may and will change incompatibly from one commit to another. 🚧
Zadig is a cloud native, distributed, developer-oriented continuous delivery product.
Zadig Developer-oriented Continuous Delivery Product English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use?
high performance distributed task scheduling system, Support multi protocol scheduling tasks
high performance distributed task scheduling system, Support multi protocol scheduling tasks
🤖🤝A tool to test and analyze storage and retrieval deal capability on the Filecoin network.
Dealbot A tool to test and analyze storage and retrieval deal capability on the Filecoin network. Getting Started Clone the repo and build: git clone
Antenna RPC is an RPC protocol for distributed computing, it's based on QUIC and Colfer. its currently an WIP.
aRPC - Antenna Remote Procedure Call Antenna remote procedure call (aRPC) is an RPC protocol focused on distributed processing and HPC. aRPC is implem
Sample Driver that provides reference implementation for Container Object Storage Interface (COSI) API
cosi-driver-minio Sample Driver that provides reference implementation for Container Object Storage Interface (COSI) API Community, discussion, contri
Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
Grafana Tempo is an open source, easy-to-use and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to ope
《蓝眼云盘》(Eyeblue Cloud Storage)
English Version 蓝眼云盘(3.1.2) 在线Demo (体验账号: demo 密码:123456) 蓝眼云盘是蓝眼开源系列代表作品之一,致力于打造精致,优雅,简约的云盘。核心功能如下: 文件列表 文件/文件夹上传 + 拖拽上传 文件分享 文件监控统计 回收站 多用户 WebDav 扫
Zadig is a cloud native, distributed, developer-oriented continuous delivery product.
Zadig Developer-oriented Continuous Delivery Product English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? Ho
Build share and run your distributed applications.
sealer[ˈsiːlər] provides the way for distributed application package and delivery based on kubernetes.
Pixie gives you instant visibility by giving access to metrics, events, traces and logs without changing code.
Pixie gives you instant visibility by giving access to metrics, events, traces and logs without changing code.
SigNoz helps developer monitor applications and troubleshoot problems in their deployed applications
SigNoz helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc. 🔥 🖥
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.
go-mysql-server go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice.
Goroutine local storage
gls Goroutine local storage IMPORTANT NOTE It is my duty to point you to https://blog.golang.org/context, which is how Google solves all of the proble
Workflow Orchestrator
Adagio - A Workflow Orchestrator This project is currently in a constant state of flux. Don't expect it to work. Thank you o/ Adagio is a workflow exe
An enhanced HTTP client for Go
Heimdall Description Installation Usage Making a simple GET request Creating a hystrix-like circuit breaker Creating a hystrix-like circuit breaker wi
Distributed RWMutex in Go
Distributed Read-Write Mutex in Go The default Go implementation of sync.RWMutex does not scale well to multiple cores, as all readers contend on the
TalariaDB is a distributed, highly available, and low latency time-series database for Presto
TalariaDB is a distributed, highly available, and low latency time-series database that stores real-time data. It's built on top of Badger DB.
An application-oriented unified storage layer for Golang.
storage An application-oriented unified storage layer for Golang. Goal Production ready High performance Vendor agnostic Features Widely services supp
💡 A Distributed and High-Performance Monitoring System. The next generation of Open-Falcon
夜莺简介 夜莺是一套分布式高可用的运维监控系统,最大的特点是混合云支持,既可以支持传统物理机虚拟机的场景,也可以支持K8S容器的场景。同时,夜莺也不只是监控,还有一部分CMDB的能力、自动化运维的能力,很多公司都基于夜莺开发自己公司的运维平台。开源的这部分功能模块也是商业版本的一部分,所以可靠性有保
Centralized Configuration System written in Golang - Spring cloud compatible
Centralized Configuration System What is Vecosy Vecosy is a configuration service exposed through REST/GRPC. Is Spring Cloud Conf compatible and also
Distributed simple and robust release management and monitoring system.
Agente Distributed simple and robust release management and monitoring system. **This project on going work. Road map Core system First worker agent M
Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.
Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.
GoVector is a vector clock logging library written in Go.
GoVector is a vector clock logging library written in Go. The vector clock algorithm is used to order events in distributed systems in the absence of a centralized clock. GoVector implements the vector clock algorithm and provides feature-rich logging and encoding infrastructure.
LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability.
LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability. LinDB stores all monitoring data of ELEME Inc, there is 88TB incremental writes per day and 2.7PB total raw data.
Inspr is an application mesh for simple, fast and secure development of distributed applications.
Inspr is an engine for running distributed applications, using multiple communication patterns such as pub sub and more, focused on type consistency a
A key-value db api with multiple storage engines and key generation
Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git
Get any cryptocurrencies ticker and trade data in real time from multiple exchanges and then save it in multiple storage systems.
Cryptogalaxy is an app which will get any cryptocurrencies ticker and trade data in real time from multiple exchanges and then saves it in multiple storage systems.
Vald. A Highly Scalable Distributed Vector Search Engine
Vald is a highly scalable distributed fast approximate nearest neighbor dense vector search engine.
Asynq: simple, reliable, and efficient distributed task queue in Go
Asynq: simple, reliable, and efficient distributed task queue in Go
Golang implementation of Sliding Window Algorithm for distributed rate limiting.
slidingwindow Golang implementation of Sliding Window Algorithm for distributed rate limiting. Installation $ go get -u github.com/RussellLuo/slidingw
Bigmachine is a library for self-managing serverless computing in Go
Bigmachine Bigmachine is a toolkit for building self-managing serverless applications in Go. Bigmachine provides an API that lets a driver process for
分布式定时任务库 distributed-cron
dcron 分布式定时任务库 原理 基于redis同步节点数据,模拟服务注册。然后将任务名 根据一致性hash 选举出执行该任务的节点。 流程图 特性 负载均衡:根据任务数据和节点数据均衡分发任务。 无缝扩容:如果任务节点负载过大,直接启动新的服务器后部分任务会自动迁移至新服务实现无缝扩容。
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is an open-source POSIX file system built on top of Redis and object storage
a Framework for creating mesh networks using technologies and design patterns of Erlang/OTP in Golang
Ergo Framework Implementation of Erlang/OTP in Golang. Up to x5 times faster than original Erlang/OTP. The easiest drop-in replacement for your hot no
go mysql driver, support distributed transaction
Go-MySQL-Driver A MySQL-Driver for Go's database/sql package Features Requirements Installation Usage DSN (Data Source Name) Password Protocol Address
Easy to use distributed event bus similar to Kafka
chukcha Easy to use distributed event bus similar to Kafka. The event bus is designed to be used as a persistent intermediate storage buffer for any k
A course to build distributed key-value service based on TiKV model
The TinyKV Course This is a series of projects on a key-value storage system built with the Raft consensus algorithm
Machinery is an asynchronous task queue/job queue based on distributed message passing.
Machinery Machinery is an asynchronous task queue/job queue based on distributed message passing. V2 Experiment First Steps Configuration Lock Broker
Celery Distributed Task Queue in Go
gocelery Go Client/Server for Celery Distributed Task Queue Why? Having been involved in several projects migrating servers from Python to Go, I have
a microservice framework for rapid development of micro services in Go with rich eco-system
中文版README Go-Chassis is a microservice framework for rapid development of microservices in Go. it focus on helping developer to deliver cloud native a
Fast Raft framework using the Redis protocol for Go
This project has been archived. Please check out Uhaha for a fitter, happier, more productive Raft framework. Finn is a fast and simple framework for
A feature complete and high performance multi-group Raft library in Go.
Dragonboat - A Multi-Group Raft library in Go / 中文版 News 2021-01-20 Dragonboat v3.3 has been released, please check CHANGELOG for all changes. 2020-03
A distributed, fault-tolerant pipeline for observability data
Table of Contents What Is Veneur? Use Case See Also Status Features Vendor And Backend Agnostic Modern Metrics Format (Or Others!) Global Aggregation
CNCF Jaeger, a Distributed Tracing Platform
Jaeger - a Distributed Tracing System Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing platform created by Uber Technologies and do
[mirror] Performance measurement, storage, and analysis.
Go performance measurement, storage, and analysis tools This subrepository holds the source for various packages and tools related to performance meas
A course to build the SQL layer of a distributed database.
TinySQL TinySQL is a course designed to teach you how to implement a distributed relational database in Go. TinySQL is also the name of the simplifed
VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.
NOTICE: This project have moved to fuse-query VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse. Features High Perform
LBADD: An experimental, distributed SQL database
LBADD Let's build a distributed database. LBADD is an experimental distributed SQL database, written in Go. The goal of this project is to build a dat
A GPU-powered real-time analytics storage and query engine.
AresDB AresDB is a GPU-powered real-time analytics storage and query engine. It features low query latency, high data freshness and highly efficient i
RadonDB is an open source, cloud-native MySQL database for building global, scalable cloud services
OverView RadonDB is an open source, Cloud-native MySQL database for unlimited scalability and performance. What is RadonDB? RadonDB is a cloud-native
Distributed reliable key-value store for the most critical data of a distributed system
etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order
Database migrations. CLI and Golang library.
migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to
Pholcus is a distributed high-concurrency crawler software written in pure golang
Pholcus Pholcus(幽灵蛛)是一款纯 Go 语言编写的支持分布式的高并发爬虫软件,仅用于编程学习与研究。 它支持单机、服务端、客户端三种运行模式,拥有Web、GUI、命令行三种操作界面;规则简单灵活、批量任务并发、输出方式丰富(mysql/mongodb/kafka/csv/excel等
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Crawlab 中文 | English Installation | Run | Screenshot | Architecture | Integration | Compare | Community & Sponsorship | CHANGELOG | Disclaimer Golang-
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).
中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不
SeaweedFS is a distributed storage system for blobs, objects, files, and data warehouse, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.
SeaweedFS Sponsor SeaweedFS via Patreon SeaweedFS is an independent Apache-licensed open source project with its ongoing development made possible ent
High Performance, Kubernetes Native Object Storage
MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag
High-Performance server for NATS, the cloud native messaging system.
NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Fo
A realtime distributed messaging platform
Source: https://github.com/nsqio/nsq Issues: https://github.com/nsqio/nsq/issues Mailing List: [email protected] IRC: #nsq on freenode Docs:
Distributed cache with gossip peer membership enrollment.
Autocache Groupcache enhanced with memberlist for distributed peer discovery. TL;DR See /_example/ for usage. Run docker-compose -f _example/docker-co
🦖 Streaming-Serverless Framework for Low-latency Edge Computing applications, running atop QUIC protocol, engaging 5G technology.
YoMo YoMo is an open-source Streaming Serverless Framework for building Low-latency Edge Computing applications. Built atop QUIC Transport Protocol an
Temporal service and CLI
Temporal Temporal is a microservice orchestration platform which enables developers to build scalable applications without sacrificing productivity or
Git-like capabilities for your object storage
What is lakeFS lakeFS is an open source layer that delivers resilience and manageability to object-storage based data lakes. With lakeFS you can build
7 days golang programs from scratch (web framework Gee, distributed cache GeeCache, object relational mapping ORM framework GeeORM, rpc framework GeeRPC etc) 7天用Go动手写/从零实现系列
7 days golang programs from scratch README 中文版本 7天用Go从零实现系列 7天能写什么呢?类似 gin 的 web 框架?类似 groupcache 的分布式缓存?或者一个简单的 Python 解释器?希望这个仓库能给你答案
Distributed Task Scheduling System|分布式定时任务调度平台
Crocodile Distributed Task Scheduling System English | 中文 Introduction A distributed task scheduling system based on Golang that supports http request
A Golang based high performance, scalable and distributed workflow framework
Go-Flow A Golang based high performance, scalable and distributed workflow framework It allows to programmatically author distributed workflow as Dire
A Distributed Continuous Integration System from MongoDB
Evergreen Evergreen is a distributed continuous integration system built by MongoDB. It dynamically allocates hosts to run tasks in parallel across ma
REST Layer, Go (golang) REST API framework
REST Layer REST APIs made easy. REST Layer is an API framework heavily inspired by the excellent Python Eve. It helps you create a comprehensive, cust
A programmable, observable and distributed job orchestration system.
📖 Overview Odin is a programmable, observable and distributed job orchestration system which allows for the scheduling, management and unattended bac
Open source Observability Platform. 👉 SigNoz helps developers find issues in their deployed applications & solve them quickly
SigNoz SigNoz is an opensource observability platform. SigNoz uses distributed tracing to gain visibility into your systems and powers data using Kafk
Multitiered file storage API built on Filecoin and IPFS
Powergate Powergate is a multitiered file storage API built on Filecoin and IPFS, and an index builder for Filecoin data. It's designed to be modular
LBADD: An experimental, distributed SQL database
LBADD Let's build a distributed database. LBADD is an experimental distributed SQL database, written in Go. The goal of this project is to build a dat
A distributed system for embedding-based retrieval
Overview Vearch is a scalable distributed system for efficient similarity search of deep learning vectors. Architecture Data Model space, documents, v
Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
Cadence Visit cadenceworkflow.io to learn about Cadence. This repo contains the source code of the Cadence server. To implement workflows, activities
GO2P is a P2P framework, designed with flexibility and simplicity in mind
go2p golang p2p framework By v-braun - viktor-braun.de. Description GO2P is a P2P framework, designed with flexibility and simplicity in mind. You can
Akutan is a distributed knowledge graph store, sometimes called an RDF store or a triple store.
Akutan is a distributed knowledge graph store, sometimes called an RDF store or a triple store. Knowledge graphs are suitable for modeling data that is highly interconnected by many types of relationships, like encyclopedic information about the world. A knowledge graph store enables rich queries on its data, which can be used to power real-time interfaces, to complement machine learning applications, and to make sense of new, unstructured information in the context of the existing knowledge.
A distributed MySQL binlog storage system built on Raft
What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl
Visualize how a projects source code is distributed among its files and folders
Source Code Visualizer Visualize the code distribution in a project. Applications Applications include: Visualizing code distribution for more educate