BlobStore is a highly reliable,highly available and ultra-large scale distributed storage system

BlobStore

Overview

BlobStore is a highly reliable,highly available and ultra-large scale distributed storage system. The system adopts Reed-Solomon code, which provides higher data durability with less storage cost than use three copies backup technology, and supports multiple erasure code modes multiple availability zones,and optimizes for small file to meet the storage needs of different scenarios. Some key features of BlobStore include:

  • ultra-large scale
  • high reliability
  • flexible deployment
  • low cost

Documents

English version: https://cubefs.readthedocs.io/en/latest/

Chinese version: https://cubefs.readthedocs.io/zh_CN/latest/

Build BlobStore

$ git clone http://github.com/cubefs/blobstore.git
$ cd blobstore
$ source env.sh
$ ./build.sh

Deploy BlobStore

For more details please refer to documentation.

Manage BlobStore

For more details please refer to documentation.

License

BlobStore is licensed under the Apache License, Version 2.0. For detail see LICENSE and NOTICE.

Owner
CubeFS
CubeFS
https://github.com/cubeFS/cubefs-blobstore
Similar Resources

The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

Nov 19, 2021

Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Dec 24, 2019

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Jan 1, 2023

💡 A Distributed and High-Performance Monitoring System. The next generation of Open-Falcon

💡 A Distributed and High-Performance Monitoring System. The next generation of Open-Falcon

夜莺简介 夜莺是一套分布式高可用的运维监控系统，最大的特点是混合云支持，既可以支持传统物理机虚拟机的场景，也可以支持K8S容器的场景。同时，夜莺也不只是监控，还有一部分CMDB的能力、自动化运维的能力，很多公司都基于夜莺开发自己公司的运维平台。开源的这部分功能模块也是商业版本的一部分，所以可靠性有保

Jan 5, 2023

A distributed and coördination-free log management system

A distributed and coördination-free log management system

OK Log is archived I hoped to find the opportunity to continue developing OK Log after the spike of its creation. Unfortunately, despite effort, no su

Dec 26, 2022

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

Jan 4, 2023

Distributed-system - Practicing and learning the foundations of DS with Go

Distributed-System For practicing and learning the foundations of distributed sy

May 4, 2022

A distributed system for embedding-based retrieval

A distributed system for embedding-based retrieval

Overview Vearch is a scalable distributed system for efficient similarity search of deep learning vectors. Architecture Data Model space, documents, v

Dec 30, 2022

a dynamic configuration framework used in distributed system

a dynamic configuration framework used in distributed system

go-archaius This is a light weight configuration management framework which helps to manage configurations in distributed system The main objective of

Dec 9, 2022
Comments
  • blocking key deadlock problem

    blocking key deadlock problem

    问题场景：

    if kl == nil {
		l.lock.RUnlock()
		l.lock.Lock()
		kl = newBlocker(l.limit)
		l.keyMap[key] = kl
		kl.addRef()
		l.lock.Unlock()
    1. In a concurrent scenario, when aquire a single key, the obtained key maybe nil and no secondary check is performed, which may result in the same key being overwritten.
    if kl.loadRef() < 0 {
			l.lock.Unlock()
			panic("internal error: refs < 0")
		}
		if kl.loadRef() == 0 {
			delete(l.keyMap, key)
		}
		kls = append(kls, kl)
	}
	for _, kl := range kls {
		kl.release()
	}
    1. When the key is released, there is a problem with the reference count judgment, resulting in the repeated release of the key's lock
Related tags
Distributed Systems cubefs-blobstore
Easy to use Raft library to make your app distributed, highly available and fault-tolerant
 Easy to use Raft library to make your app distributed, highly available and fault-tolerant

An easy to use customizable library to make your Go application Distributed, Highly available, Fault Tolerant etc... using Hashicorp's Raft library wh

Nov 16, 2022
SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.
 SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.

SeaweedFS Sponsor SeaweedFS via Patreon SeaweedFS is an independent Apache-licensed open source project with its ongoing development made possible ent

Jan 5, 2023
Ultra performant API Gateway with middlewares
 Ultra performant API Gateway with middlewares

The KrakenD framework An open framework to assemble ultra performance API Gateways with middlewares; core service of the KrakenD API Gateway. Looking

Dec 29, 2022
Asynq: simple, reliable, and efficient distributed task queue in Go
 Asynq: simple, reliable, and efficient distributed task queue in Go

Asynq: simple, reliable, and efficient distributed task queue in Go

Dec 31, 2022
A distributed MySQL binlog storage system built on Raft
 A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

Dec 31, 2022
A distributed key-value storage system developed by Alibaba Group

Product Overview Tair is fast-access memory (MDB)/persistent (LDB) storage service. Using a high-performance and high-availability distributed cluster

Dec 31, 2022
Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Distributed-Services This project is essentially a result of my attempt to under

Jun 1, 2022
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
 Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

glow Purpose Glow is providing a library to easily compute in parallel threads or distributed to clusters of machines. This is written in pure Go. I a

Dec 30, 2022
Distributed disk storage database based on Raft and Redis protocol.
 Distributed disk storage database based on Raft and Redis protocol.

IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and

Dec 31, 2022
A zero cost, faster multi-language bidirectional microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! 𝐉𝐚𝐯𝐚有𝐝𝐮𝐛𝐛𝐨, 𝐆𝐨𝐥𝐚𝐧𝐠有𝐫𝐩𝐜𝐱!
 A zero cost, faster multi-language bidirectional microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! 𝐉𝐚𝐯𝐚有𝐝𝐮𝐛𝐛𝐨, 𝐆𝐨𝐥𝐚𝐧𝐠有𝐫𝐩𝐜𝐱!

Official site: http://rpcx.io Notice: etcd etcd plugin has been moved to rpcx-etcd Announce A tcpdump-like tool added: rpcxdump。 You can use it to deb

Jan 7, 2023