215 Resources
Golang storage Libraries
Kalasa is a NoSQL database and provides more data structures for ease of use.
Kalasa Kalasa is a NoSQL database and provides more data structures for ease of use. 简体中文 | English 特 性 嵌入的存储引擎 数据可以加密存储 可以自定义实现存储加密器 即使数据文件被拷贝，也保证存储数
A version control system to manage large files.
ArtiVC ArtiVC (Artifacts Version Control) is a handy command-line tool for data versioning on cloud storage. With only one command, it helps you neatl
Local Storage is one of HwameiStor components. It will provision the local LVM volume.
Local Storage Module English | Simplified_Chinese Introduction Local Storage is one of modules of HwameiStor which is a cloud native local storage sys
Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
Grafana Mimir Grafana Mimir is an open source software project that provides a scalable long-term storage for Prometheus. Some of the core strengths o
ArcticDB - an embeddable columnar database written in Go
This project is still in its infancy, consider it not production-ready, probably has various consistency and correctness problems and all API will cha
Kitten is a distributed file system optimized for small file storage, inspired by Facebook's Haystack.
Kitten is a distributed file system optimized for small file storage, inspired by Facebook's Haystack.
Mini file storage with Go (Golang)
#Microstorage This is my exercise of creating simple file storage with GoLang Purpose: store and manipulate with user`s images in my pet projects ##Ve
azqlite is a lightweight wrapper around Azure's SDK to interact with the Azure Storage Queue service in a simpler and more idiomatic way.
azqlite azqlite is a lightweight wrapper around github.com/Azure/azure-storage-queue-go to interact with the Azure Storage Queue service in a simpler
Verifier - Golang JWT token verifier with storage(default Redis)
verifier Golang JWT token verifier with storage(default Redis) Usage go get -u g
Jaeger-influxdb - The repository that contains InfluxDB Storage gRPC plugin for Jaeger
NOTICE: This repository is archived and is no longer maintained. Please use http
Users - Restful API that can get/create/update/delete user data from persistence storage
Users Service The users service is used to maintain Restful API that can get/cre
Querycrate - A simple library that loads and keeps SQL queries from files
QueryCrate A simple library for loading & getting string queries from files. How
Golang REST Layer SQL Storage Handler
This REST Layer resource storage backend stores data in a SQL Database using database/sql.
LotsusDB is a fast k/v database compatible with LSM tree and B+ tree
LotusDB LotsusDB is a fast k/v database compatible with LSM tree and B+ tree. Key features: Combine the advantages of LSM and B+ tree Fast read/write
SMART information of local storage devices as Prometheus metrics
hpessa-exporter Overview Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using
A Go implementation of an in-memory bloom filter, with support for boltdb and badgerdb as optional data persistent storage.
Sprout A bloom filter is a probabilistic data structure that is used to determine if an element is present in a set. Bloom filters are fast and space
Asset storage and on-the-fly image resize
air - Asset & Image Resize Asset storage and on-the-fly image resize powered by libvips. Uploading an asset $ http -f POST http://127.0.0.1:1323/uploa
Every 10 minutes, memory, cpu and storage usage is checked and if they over 80%, sending alert via email.
linux-alert Every 10 minutes, memory, cpu and storage usage is checked and if they over 80%, sending alert via email. Usage Create .env file from .env
BlobStore is a highly reliable,highly available and ultra-large scale distributed storage system
BlobStore Overview Documents Build BlobStore Deploy BlobStore Manage BlobStore License Overview BlobStore is a highly reliable,highly available and ul
Descentralized object storage network for resilient services
Polar Descentralized object storage network for resilient services Polar is a binary that you can run on any linux distribution, it will expose a RPC
The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your container orchestrator
fortress-csi The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your co
Jaeger-s3 - Jaeger gRPC storage plugin for Amazon S3
jaeger-s3 jaeger-s3 is gRPC storage plugin for Jaeger, which uses Amazon Kinesis
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm
The TinyKV Course The TinyKV course builds a key-value storage system with the R
Log to cloud object storage for golang. Zap extension.
cos-logger Log to cloud object storage for golang implemented as io.Writer. Use it as a plugin/extension to uber-go/zap logger Configure logger and ad
A simple image hosting script in Golang (Smart Storage with Telegram Cloud Storage)
image-upload-tg It's a simple image hosting script in Golang. It's have http server and image can be uploaded from here, also images store temporary i
Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using HPE Smart Storage Administrator tool
hpessa-exporter Overview Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using
Keyval - A simple key-value storage library written in Go
keyval keyval is a simple key-value storage library written in Go and its back c
ttlcache – is a simple and efficient in-memory key value storage with TTL for each record
ttlcache About ttlcache – is a simple and efficient in-memory key value storage with TTL for each record. The key is of uint64 type. Library provides
A letsencrypt client that uses etcd as its storage.
letsencrypt-with-etcd This is a letsencrypt client that uses etcd as its storage. It stores your (automatically created) LetsEncrypt account in /letse
This service is intented to collect data using grpc using Go lang backend and cassandra DB as storage
Go Data Collection This service is intented to collect data using grpc using Go lang backend and cassandra DB as storage. Dev Setup make test_env make
A pure Golang implementation of Rockchip rknand vendor storage interface.
go-rkvendorstorage A pure Golang implementation of Rockchip rknand vendor storage interface. Usage package main import ( "fmt" "github.com/jamesits
A cross goroutine storage tool with very simple implementation and function.
Simple-goroutine-local is a cross goroutine storage tool with very simple implementation and function (the concept is similar to Java ThreadLocal). Ge
Simple and minimal image server capable of storing, resizing, converting and caching images.
webp-server Simple and minimal image server capable of storing, resizing, converting, and caching images. You can quickly find out how it works by loo
REST api using fiber framework written in golang and using firebase ecosystem to authentication, storage and firestore as a db and use clean architecture as base
Backend API Example FiberGo Framework Docs : https://github.com/gofiber Info This application using firebase ecosystem Firebase Auth Cloud Storage Fir
Imaginarium - A simple golang image storage engine
Imaginarium A simple golang image storage engine. Used to create and store diffe
Flash-metrics - Flash Metrics Storage With Golang
Flash Metrics Storage bootstrap: $ echo -e "max-index-length = 12288" tidb.con
Simple CLI interface to in-memory key/value storage a la redis.
Simple memory key value Simple CLI interface to in-memory key/value storage a la redis. Running Can be run through go directly or through docker using
Go-storage - A vendor-neutral storage library for Golang: Write once, run on every storage service.
go-storage Website | Documentation | Community A vendor-neutral storage library for Golang. Vision Write once, run on every storage service. Goal Vend
OVO is an In-Memory Distributed Cache and a Key/Value Storage.
OVO is an In-Memory Distributed Cache and a Key/Value Storage. Main features OVO is a distributed in-memory cache that supports data sharding on m
Contentrouter - Protect static content via Firebase Hosting with Cloud Run and Google Cloud Storage
contentrouter A Cloud Run service to gate static content stored in Google Cloud
A collection about awesome blockchains
A collection about awesome blockchains - open distributed public databases w/ crypto hashes incl. git ;-). Blockchains are the new tulips :tulip::tulip::tulip:. Distributed is the new centralized.
Vulcan extends Prometheus adding horizontal scalability and long-term storage
Warning: This project is currently not maintained, and there is no plan to do so ATM. Vulcan Vulcan extends Prometheus adding horizontal scalability a
Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
Overview Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added
A time series storage engine for Go
catena n. A closely linked series. Catena is a storage engine for time series data. Documentation Extensive documentation is not available yet, but th
Go client for Apptainer key storage and retrieval using HKP
Container Key Client This project provides a Go client to Apptainer for key stor
🚀A fast, stable and embedded k-v storage in pure Golang, supports string, list, hash, set, sorted set.
English | 简体中文 rosedb is not production-ready, we are doing more test now. rosedb is a fast, stable and embedded key-value (k-v) storage engine based
Secure SDK/vault for personal records/PII built to comply with GDPR
Databunker Databunker is a network-based, self-hosted, GDPR compliant, secure vault for personal data or PII: https://databunker.org/ Project demo is
A custom IPFS/Filecoin node that makes it easy to pin IPFS content and make Filecoin deals.
Estuary An experimental ipfs node Building Requirements: go (1.15 or higher) jq hwloc opencl Run make clean all inside the estuary directory Running T
A server for TurboRepo Remote Cache to store cache artefacts in Google Cloud Storage or Amazon S3
Tapico Turborepo Remote Cache This is an implementation of Vercel's Turborepo Remote Cache API endpoints used by the turborepo CLI command. This solut
BTFS - The First Scalable Decentralized Storage System - A Foundational Platform for Decentralized Applications
go-btfs What is BTFS? BitTorrent File System (BTFS) is a protocol forked from IPFS that utilizes the TRON network and the BitTorrent Ecosystem for int
A High Performance Object Storage released under Apache License
MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag
QingStor Object Storage service support for go-storage
go-services-qingstor QingStor Object Storage service support for go-storage. Install go get github.com/minhjh/go-service-qingstor/v3 Usage import ( "
💾 Wolke API is the API behind Wolke image storage and processing aswell as user management
💾 Wolke API Wolke API is the API behind Wolke image storage and processing aswell as user management Deploying To deploy Wolke Bot you'll need podman
MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.
MinIO Client Quickstart Guide MinIO Client (mc) provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff, find etc. It supports f
🍔 Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result.
🍔 Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result. The server transfer request to a third-party resource for .csv-file uploading and saves the products to own database.
a wrapper around BadgerDB providing a simple API.
Carbon Cache A wrapper around BadgerDB providing a simple API. NOTE This package is provided "as is" with no guarantee. Use it at your own risk and al
csg ("Credential Storage with Go") - a tool to organize the storage of credentials found during a CTF or Pentest.
csg csg ("Credential Storage with Go") - a tool to organize the storage of credentials found during a CTF or Pentest. Check out my blog on csg for mor
Google Drive client for the commandline
drive drive is a tiny program to pull or push Google Drive files. drive was originally developed by Burcu Dogan while working on the Google Drive team
CLI tool to upload object to s3-compatible storage backend and set download policy for it.
typora-s3 CLI tool to upload object to s3-compatible storage backend and set download policy for it. Build $ git clone https://github.com/fengxsong/ty
Sugared logger for console with request id
📝 zlog You don't know what it's like, you don't have a clue If you did you'd find yourselves doing the same thing too Sugared logger for console with
Linux GUI viewer for ZFS pool, dataset and host storage.
This is a WIP in alpha stage. zgui Linux GUI viewer for ZFS pool, dataset and host storage. zgui used libzfs directly and not ZFS command line tools.
GoDrive: A cloud storage system similar to Dropbox or Google Drive, with resilient
Cloud Storage Service Author: Marisa Tania, Ryan Tjakrakartadinata Professor: Matthew Malensek See project spec here: https://www.cs.usfca.edu/~mmalen
A probabilistic data structure service and storage
Skizze ([ˈskɪt͡sə]: german for sketch) is a sketch data store to deal with all problems around counting and sketching using probabilistic data-structu
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values. Authors: Sanjay Ghem
The Swift Virtual File System
*** This project is not maintained anymore *** The Swift Virtual File System SVFS is a Virtual File System over Openstack Swift built upon fuse. It is
SeaweedFS a fast distributed storage system for blobs, objects, files, and data lake, for billions of files
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go Overview Goofys allows you to mount an S3 bucket as a filey system. It's a
A user-space file system for interacting with Google Cloud Storage
gcsfuse is a user-space file system for interacting with Google Cloud Storage. Current status Please treat gcsfuse as beta-quality software. Use it fo
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.
Distributed disk storage database based on Raft and Redis protocol.
IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and
Cloud-Native distributed storage built on and for Kubernetes
Longhorn Build Status Engine: Manager: Instance Manager: Share Manager: Backing Image Manager: UI: Test: Release Status Release Version Type 1.1 1.1.2
Storage Orchestration for Kubernetes
What is Rook? Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for a diverse se
s3git: git for Cloud Storage. Distributed Version Control for Data.
s3git: git for Cloud Storage. Distributed Version Control for Data. Create decentralized and versioned repos that scale infinitely to 100s of millions of files. Clone huge PB-scale repos on your local SSD to make changes, commit and push back. Oh yeah, it dedupes too and offers directory versioning.
Perkeep (née Camlistore) is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content.
Perkeep is your personal storage system. It's a way to store, sync, share, import, model, and back up content. Keep your stuff for life. For more, see
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr
High Performance, Kubernetes Native Object Storage
MinIO Quickstart Guide MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Ama
Dynamically provisioning persistent local storage with Kubernetes
Local Path Provisioner Overview Local Path Provisioner provides a way for the Kubernetes users to utilize the local storage in each node. Based on the
Distributed disk storage database based on Raft and Redis protocol.
IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and
The TinyKV course builds a key-value storage system with the Raft consensus algorithm
The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.
[WIP] Cheap, portable and secure NAS based on the Raspberry Pi Zero - with encryption, backups, and more
PortaDisk - Affordable Raspberry Pi Portable & Secure NAS Project Project Status: Early work in progress. web-unlock is still not ready for production
A gateway to expose s3 standard API for any storage type.
s3-gateway this project is used to be compatible with any other storage type. there are tons of object storage services in the internat. However, many
Paxoskv: a Naive and Basic paxos kv storage
paxoskv: a Naive and Basic paxos kv storage 这个repo 目前仅是用于学习的实例代码. 这是一个基于paxos, 只有200行代码的kv存储系统的简单实现, 以最简洁的形式展示paxos如何运行, 作为 可靠分布式系统-paxos的直观解释 这篇教程中的代
This is a simple file storage server. User can upload file, delete file and list file on the server.
Simple File Storage Server This is a simple file storage server. User can upload file, delete file and list file on the server. If you want to build a
rgkv is a distributed kv storage service using raft consensus algorithm.
rgkv rgkv is a distributed kv storage service using raft consensus algorithm. Get/put/append operation High Availability Sharding Linearizability Tabl
A key-value storage transaction interpretator.
kv-txn-interpreter A key-value storage transaction interpreter, which provides an etcd-like transaction interface to help you build a transaction over
K8s local storage sync for stateful set's using microk8s-hostpath storage classe
Local Storage Sync for microk8s-hostpath The goal is to be able to sync stateful sets between the different nodes of a cluster to allow the data to be
Network connecter for storage.
database Quick fast connection to database use gorm. Installation $ go get -u github.com/coolstina/connecter Example package main import ( "fmt" "
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.
The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.
PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage.
中文 | English PaddleDTX PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage. It solves the d
Void is a zero storage cost large file sharing system.
void void is a zero storage cost large file sharing system. License Copyright © 2021 Changkun Ou. All rights reserved. Unauthorized using, copying, mo
CSI for S3 compatible SberCloud Object Storage Service
sbercloud-csi-obs CSI for S3 compatible SberCloud Object Storage Service This is a Container Storage Interface (CSI) for S3 (or S3 compatible) storage
CSI driver NVMf mainly supports RDMA and TCP for Software Defined Storage by NVMf
CSI NVMf driver Overview This is a repository for NVMe-oF CSI Driver. Currently it implements bare minimum of th CSI spec. Requirements The CSI NVMf d
Key-Value Storage written in Go.
kvs kvs is an in-memory key-value storage written in Go. It has 2 different usage. It can be used as a package by importing it to your code or as a se
Store files in Telegram messages for free and access them from a modern Web UI
Telegram Storage Store files in Telegram messages for free and access them in a nice web UI. Telegram allows to store files (of max 2GB each) for a un
A simple RESTful API for storing and retrieving data during MRI image reconstructions
MRD Storage Server The MRD Storage Server provides a simple RESTful API for storing and retrieving data during MRI image reconstructions. It supports:
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.
The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.
Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.
Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr