259 Resources
Golang object-storage Libraries
Kalasa is a NoSQL database and provides more data structures for ease of use.
Kalasa Kalasa is a NoSQL database and provides more data structures for ease of use. 简体中文 | English 特 性 嵌入的存储引擎 数据可以加密存储 可以自定义实现存储加密器 即使数据文件被拷贝，也保证存储数
A version control system to manage large files.
ArtiVC ArtiVC (Artifacts Version Control) is a handy command-line tool for data versioning on cloud storage. With only one command, it helps you neatl
Local Storage is one of HwameiStor components. It will provision the local LVM volume.
Local Storage Module English | Simplified_Chinese Introduction Local Storage is one of modules of HwameiStor which is a cloud native local storage sys
Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
Grafana Mimir Grafana Mimir is an open source software project that provides a scalable long-term storage for Prometheus. Some of the core strengths o
ArcticDB - an embeddable columnar database written in Go
This project is still in its infancy, consider it not production-ready, probably has various consistency and correctness problems and all API will cha
💧 Visual Data Preparation (VDP) is an open-source tool to seamlessly integrate Vision AI with the modern data stack
Website | Community | Blog Get Early Access Visual Data Preparation (VDP) is an open-source tool to streamline the end-to-end visual data processing p
Kitten is a distributed file system optimized for small file storage, inspired by Facebook's Haystack.
Kitten is a distributed file system optimized for small file storage, inspired by Facebook's Haystack.
Mini file storage with Go (Golang)
#Microstorage This is my exercise of creating simple file storage with GoLang Purpose: store and manipulate with user`s images in my pet projects ##Ve
azqlite is a lightweight wrapper around Azure's SDK to interact with the Azure Storage Queue service in a simpler and more idiomatic way.
azqlite azqlite is a lightweight wrapper around github.com/Azure/azure-storage-queue-go to interact with the Azure Storage Queue service in a simpler
Tapestry is an underlying distributed object location and retrieval system (DOLR) which can be used to store and locate objects. This distributed system provides an interface for storing and retrieving key-value pairs.
Tapestry This project implements Tapestry, an underlying distributed object location and retrieval system (DOLR) which can be used to store and locate
SFTP backed by LDAP and S3-compatible object stores
RainSFTP RainSFTP is an implementaion of the Secure File Transfer Protocol backed by LDAP for authentication and an S3-compatible object store. This m
Verifier - Golang JWT token verifier with storage(default Redis)
verifier Golang JWT token verifier with storage(default Redis) Usage go get -u g
Jose - JavaScript Object Signing and Encryption (JOSE)
jose JavaScript Object Signing and Encryption JOSE implemented in Go. RFCs RFC75
Jaeger-influxdb - The repository that contains InfluxDB Storage gRPC plugin for Jaeger
NOTICE: This repository is archived and is no longer maintained. Please use http
Users - Restful API that can get/create/update/delete user data from persistence storage
Users Service The users service is used to maintain Restful API that can get/cre
Querycrate - A simple library that loads and keeps SQL queries from files
QueryCrate A simple library for loading & getting string queries from files. How
Golang REST Layer SQL Storage Handler
This REST Layer resource storage backend stores data in a SQL Database using database/sql.
LotsusDB is a fast k/v database compatible with LSM tree and B+ tree
LotusDB LotsusDB is a fast k/v database compatible with LSM tree and B+ tree. Key features: Combine the advantages of LSM and B+ tree Fast read/write
Deep walk by object with reflection
Deep walk by object with reflection. Walker.Walk(v interface{}) call callback function for v object, every field if it struct, every item for slice/ar
SMART information of local storage devices as Prometheus metrics
hpessa-exporter Overview Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using
A Go implementation of an in-memory bloom filter, with support for boltdb and badgerdb as optional data persistent storage.
Sprout A bloom filter is a probabilistic data structure that is used to determine if an element is present in a set. Bloom filters are fast and space
Asset storage and on-the-fly image resize
air - Asset & Image Resize Asset storage and on-the-fly image resize powered by libvips. Uploading an asset $ http -f POST http://127.0.0.1:1323/uploa
Every 10 minutes, memory, cpu and storage usage is checked and if they over 80%, sending alert via email.
linux-alert Every 10 minutes, memory, cpu and storage usage is checked and if they over 80%, sending alert via email. Usage Create .env file from .env
Go object files
Go object files C developers will be familiar with compiling C source files (.c) into object files (.o) before linking them into their final form. Did
BlobStore is a highly reliable,highly available and ultra-large scale distributed storage system
BlobStore Overview Documents Build BlobStore Deploy BlobStore Manage BlobStore License Overview BlobStore is a highly reliable,highly available and ul
Descentralized object storage network for resilient services
Polar Descentralized object storage network for resilient services Polar is a binary that you can run on any linux distribution, it will expose a RPC
The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your container orchestrator
fortress-csi The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your co
Jaeger-s3 - Jaeger gRPC storage plugin for Amazon S3
jaeger-s3 jaeger-s3 is gRPC storage plugin for Jaeger, which uses Amazon Kinesis
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm
The TinyKV Course The TinyKV course builds a key-value storage system with the R
Log to cloud object storage for golang. Zap extension.
cos-logger Log to cloud object storage for golang implemented as io.Writer. Use it as a plugin/extension to uber-go/zap logger Configure logger and ad
A simple image hosting script in Golang (Smart Storage with Telegram Cloud Storage)
image-upload-tg It's a simple image hosting script in Golang. It's have http server and image can be uploaded from here, also images store temporary i
Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using HPE Smart Storage Administrator tool
hpessa-exporter Overview Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using
Keyval - A simple key-value storage library written in Go
keyval keyval is a simple key-value storage library written in Go and its back c
ttlcache – is a simple and efficient in-memory key value storage with TTL for each record
ttlcache About ttlcache – is a simple and efficient in-memory key value storage with TTL for each record. The key is of uint64 type. Library provides
A letsencrypt client that uses etcd as its storage.
letsencrypt-with-etcd This is a letsencrypt client that uses etcd as its storage. It stores your (automatically created) LetsEncrypt account in /letse
This service is intented to collect data using grpc using Go lang backend and cassandra DB as storage
Go Data Collection This service is intented to collect data using grpc using Go lang backend and cassandra DB as storage. Dev Setup make test_env make
A simple embeddable scripting language which supports concurrent event processing.
ECAL ECAL is an ECA (Event Condition Action) language for concurrent event processing. ECAL can define event-based systems using rules which are trigg
A pure Golang implementation of Rockchip rknand vendor storage interface.
go-rkvendorstorage A pure Golang implementation of Rockchip rknand vendor storage interface. Usage package main import ( "fmt" "github.com/jamesits
A cross goroutine storage tool with very simple implementation and function.
Simple-goroutine-local is a cross goroutine storage tool with very simple implementation and function (the concept is similar to Java ThreadLocal). Ge
Simple and minimal image server capable of storing, resizing, converting and caching images.
webp-server Simple and minimal image server capable of storing, resizing, converting, and caching images. You can quickly find out how it works by loo
A logging library with strack traces, object dumping and optional timestamps
go-log This is a Golang library with logging related functions which I use in my different projects. Usage package main import ( "github.com/piet
REST api using fiber framework written in golang and using firebase ecosystem to authentication, storage and firestore as a db and use clean architecture as base
Backend API Example FiberGo Framework Docs : https://github.com/gofiber Info This application using firebase ecosystem Firebase Auth Cloud Storage Fir
Imaginarium - A simple golang image storage engine
Imaginarium A simple golang image storage engine. Used to create and store diffe
Flash-metrics - Flash Metrics Storage With Golang
Flash Metrics Storage bootstrap: $ echo -e "max-index-length = 12288" tidb.con
Light Object-Relational Environment (LORE) provides a simple and lightweight pseudo-ORM/pseudo-struct-mapping environment for Go
LORE Light Object-Relational Environment (LORE) provides a simple and lightweight pseudo-ORM/pseudo-struct-mapping environment for Go. Motivation With
Reads JSON object (stream) from file/stdin and routes it/them to GCP Pub/Sub topics.
json2pubsub Publish JSON object (stream) into GCP Pub/Sub topic based on a field value. Usage: json2pubsub --project=STRING mapping ... Reads JSON
Simple CLI interface to in-memory key/value storage a la redis.
Simple memory key value Simple CLI interface to in-memory key/value storage a la redis. Running Can be run through go directly or through docker using
State observer - StateObserver used to synchronize the local(cached) state of the remote object with the real state
state observer StateObserver used to synchronize the local(cached) state of the
S3pd - CLI utility that downloads multiple s3 objects at a time, with multiple range-requests issued per object
S3 Parallel Downloader CLI utility that downloads multiple s3 objects at a time,
Go-storage - A vendor-neutral storage library for Golang: Write once, run on every storage service.
go-storage Website | Documentation | Community A vendor-neutral storage library for Golang. Vision Write once, run on every storage service. Goal Vend
OVO is an In-Memory Distributed Cache and a Key/Value Storage.
OVO is an In-Memory Distributed Cache and a Key/Value Storage. Main features OVO is a distributed in-memory cache that supports data sharding on m
Ots - The Bhojpur OTS is a software-as-a-service product used as an Object Tracking System based on Bhojpur.NET Platform for application delivery.
Bhojpur OTS - Object Tracking System The Bhojpur OTS is a software-as-a-service product used as an Object Tracking System based on Bhojpur.NET Platfor
Contentrouter - Protect static content via Firebase Hosting with Cloud Run and Google Cloud Storage
contentrouter A Cloud Run service to gate static content stored in Google Cloud
A collection about awesome blockchains
A collection about awesome blockchains - open distributed public databases w/ crypto hashes incl. git ;-). Blockchains are the new tulips :tulip::tulip::tulip:. Distributed is the new centralized.
Vulcan extends Prometheus adding horizontal scalability and long-term storage
Warning: This project is currently not maintained, and there is no plan to do so ATM. Vulcan Vulcan extends Prometheus adding horizontal scalability a
Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
Overview Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added
A time series storage engine for Go
catena n. A closely linked series. Catena is a storage engine for time series data. Documentation Extensive documentation is not available yet, but th
A tiny date object in Go. Tinydate uses only 4 bytes of memory
go-tinydate A tiny date object in Go. Tinydate uses 4 bytes of memory vs the 24 bytes of a standard time.Time{} A tinydate only has day precision. It
Go client for Apptainer key storage and retrieval using HKP
Container Key Client This project provides a Go client to Apptainer for key stor
🚀A fast, stable and embedded k-v storage in pure Golang, supports string, list, hash, set, sorted set.
English | 简体中文 rosedb is not production-ready, we are doing more test now. rosedb is a fast, stable and embedded key-value (k-v) storage engine based
Secure SDK/vault for personal records/PII built to comply with GDPR
Databunker Databunker is a network-based, self-hosted, GDPR compliant, secure vault for personal data or PII: https://databunker.org/ Project demo is
Cap'n Proto library and code generator for Go
Cap'n Proto bindings for Go Cap’n Proto is an insanely fast data interchange format similar to Protocol Buffers, but much faster. It also includes a s
Removes unnecessarily saved git objects to optimize the size of the .git directory.
Git Repo Cleaner Optimizes the size of the .git directory by removing all of the files that are unnecessarily-still-saved as part of the git history.
A custom IPFS/Filecoin node that makes it easy to pin IPFS content and make Filecoin deals.
Estuary An experimental ipfs node Building Requirements: go (1.15 or higher) jq hwloc opencl Run make clean all inside the estuary directory Running T
A server for TurboRepo Remote Cache to store cache artefacts in Google Cloud Storage or Amazon S3
Tapico Turborepo Remote Cache This is an implementation of Vercel's Turborepo Remote Cache API endpoints used by the turborepo CLI command. This solut
BTFS - The First Scalable Decentralized Storage System - A Foundational Platform for Decentralized Applications
go-btfs What is BTFS? BitTorrent File System (BTFS) is a protocol forked from IPFS that utilizes the TRON network and the BitTorrent Ecosystem for int
A High Performance Object Storage released under Apache License
MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag
QingStor Object Storage service support for go-storage
go-services-qingstor QingStor Object Storage service support for go-storage. Install go get github.com/minhjh/go-service-qingstor/v3 Usage import ( "
💾 Wolke API is the API behind Wolke image storage and processing aswell as user management
💾 Wolke API Wolke API is the API behind Wolke image storage and processing aswell as user management Deploying To deploy Wolke Bot you'll need podman
MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.
MinIO Client Quickstart Guide MinIO Client (mc) provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff, find etc. It supports f
🍔 Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result.
🍔 Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result. The server transfer request to a third-party resource for .csv-file uploading and saves the products to own database.
requestgen generates the cascade call for your request object
requestgen requestgen generates the cascade call for your request object Installation go get github.com/c9s/requestgen Usage requestgen scans all the
a wrapper around BadgerDB providing a simple API.
Carbon Cache A wrapper around BadgerDB providing a simple API. NOTE This package is provided "as is" with no guarantee. Use it at your own risk and al
csg ("Credential Storage with Go") - a tool to organize the storage of credentials found during a CTF or Pentest.
csg csg ("Credential Storage with Go") - a tool to organize the storage of credentials found during a CTF or Pentest. Check out my blog on csg for mor
Google Drive client for the commandline
drive drive is a tiny program to pull or push Google Drive files. drive was originally developed by Burcu Dogan while working on the Google Drive team
CLI tool to upload object to s3-compatible storage backend and set download policy for it.
typora-s3 CLI tool to upload object to s3-compatible storage backend and set download policy for it. Build $ git clone https://github.com/fengxsong/ty
Sugared logger for console with request id
📝 zlog You don't know what it's like, you don't have a clue If you did you'd find yourselves doing the same thing too Sugared logger for console with
Linux GUI viewer for ZFS pool, dataset and host storage.
This is a WIP in alpha stage. zgui Linux GUI viewer for ZFS pool, dataset and host storage. zgui used libzfs directly and not ZFS command line tools.
GoDrive: A cloud storage system similar to Dropbox or Google Drive, with resilient
Cloud Storage Service Author: Marisa Tania, Ryan Tjakrakartadinata Professor: Matthew Malensek See project spec here: https://www.cs.usfca.edu/~mmalen
A probabilistic data structure service and storage
Skizze ([ˈskɪt͡sə]: german for sketch) is a sketch data store to deal with all problems around counting and sketching using probabilistic data-structu
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values. Authors: Sanjay Ghem
The Swift Virtual File System
*** This project is not maintained anymore *** The Swift Virtual File System SVFS is a Virtual File System over Openstack Swift built upon fuse. It is
SeaweedFS a fast distributed storage system for blobs, objects, files, and data lake, for billions of files
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go Overview Goofys allows you to mount an S3 bucket as a filey system. It's a
A user-space file system for interacting with Google Cloud Storage
gcsfuse is a user-space file system for interacting with Google Cloud Storage. Current status Please treat gcsfuse as beta-quality software. Use it fo
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.
Distributed disk storage database based on Raft and Redis protocol.
IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and
Cloud-Native distributed storage built on and for Kubernetes
Longhorn Build Status Engine: Manager: Instance Manager: Share Manager: Backing Image Manager: UI: Test: Release Status Release Version Type 1.1 1.1.2
Storage Orchestration for Kubernetes
What is Rook? Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for a diverse se
s3git: git for Cloud Storage. Distributed Version Control for Data.
s3git: git for Cloud Storage. Distributed Version Control for Data. Create decentralized and versioned repos that scale infinitely to 100s of millions of files. Clone huge PB-scale repos on your local SSD to make changes, commit and push back. Oh yeah, it dedupes too and offers directory versioning.
Perkeep (née Camlistore) is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content.
Perkeep is your personal storage system. It's a way to store, sync, share, import, model, and back up content. Keep your stuff for life. For more, see
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr
High Performance, Kubernetes Native Object Storage
MinIO Quickstart Guide MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Ama
Dynamically provisioning persistent local storage with Kubernetes
Local Path Provisioner Overview Local Path Provisioner provides a way for the Kubernetes users to utilize the local storage in each node. Based on the
Distributed disk storage database based on Raft and Redis protocol.
IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and
The TinyKV course builds a key-value storage system with the Raft consensus algorithm
The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.