An application-oriented unified storage layer for Golang.

storage

Build Status Go dev License go storage dev

An application-oriented unified storage layer for Golang.

Goal

  • Production ready
  • High performance
  • Vendor agnostic

Features

Widely services support

Servicer operation support

  • List: list all Storager in service
  • Get: get a Storager via name
  • Create: create a Storager
  • Delete: delete a Storager

Storager operation support

Basic operations

  • Metadata: get storager metadata
  • Read: read file content
  • Write: write content into file
  • Stat: get file's metadata
  • Delete: delete a file or directory
  • List: list file in prefix or dir styles

Extended operations

  • Copy: copy a file inside storager
  • Move: move a file inside storager
  • Reach: generate a public accessible url

Multi object modes support

  • Multipart: allow doing multipart uploads
  • Append: allow appending to an object
  • Block: allow combining an object with block ids.
  • Page: allow doing random writes

Object metadata support

Common metadata

  • id: unique key in service
  • name: relative path towards service's work dir
  • type: object type cloud be file, dir, link or unknown

Optional metadata

  • size: object's content size.
  • updated-at: object's last updated time.
  • content-md5: md5 digest as defined in rfc2616
  • content-type: media type as defined in rfc2616
  • etag: entity tag as defined in rfc2616
  • storage-class: object's storage class as defined in storage proposal

Quick Start

package main

import (
	"bytes"
	"log"

	"github.com/aos-dev/go-storage/v3/pairs"
	"github.com/aos-dev/go-service-fs/v2"
)

func main() {
	// Init a service.
	store, err := fs.NewStorager(pairs.WithWorkDir("/tmp"))
	if err != nil {
		log.Fatalf("service init failed: %v", err)
	}

	content := []byte("Hello, world!")
	length := int64(len(content))
	r := bytes.NewReader(content)

	_, err = store.Write("hello", r, length)
	if err != nil {
		log.Fatalf("write failed: %v", err)
	}

	var buf bytes.Buffer

	_, err = store.Read("hello", &buf)
	if err != nil {
		log.Fatalf("storager read: %v", err)
	}

	log.Printf("%s", buf.String())
}

Examples

All examples are maintained in https://github.com/aos-dev/go-storage-example.

Sponsor

Owner
Comments
  • Idea: Notification (CDC) Support

    Idea: Notification (CDC) Support

    Our Storage Service may support sending notifications to let users get the changes of storage.

    This feature likes CDC(Change Data Capture) for DBMS.

    We may need to:

    • Define Change/Event struct that fit all storage services need
      • S3 Notification
        • https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html
      • Filesystem
        • inotify
        • use eBPF to capture changes.
          • https://github.com/cilium/ebpf ?
    • Support different event receiver
      • Work as a webhook?
      • Message Queue?
  • Tracking issues of go-storage 2021Q3 Roadmap

    Tracking issues of go-storage 2021Q3 Roadmap

    This issue will track tasks that split from BeyondStorage 2021Q3 Roadmap

    Make implemented services stable

    Steps

    • Implement features required by Storager
    • Passed integration tests.
    • Add docs and examples.
    • Update docs in go-storage and site.

    Services

    • [x] ftp: FTP
    • [x] gdrive: Google Drive
    • [x] hdfs: Hadoop Distributed File System
    • [x] ipfs: InterPlanetary File System
    • [x] memory: data that only in memory
    • [x] minio: MinIO
    • [ ] onedrive: Microsoft OneDrive
    • [ ] storj: StorJ
    • [ ] webdav: WebDAV

    Implement more services

    We expect to implement 9 of them in 2021Q3.

    • [ ] ocios: https://github.com/beyondstorage/go-storage/issues/692
    • [ ] glusterfs
    • [x] cephfs: https://github.com/beyondstorage/go-service-cephfs
    • [x] azfile: https://github.com/beyondstorage/go-service-azfile
    • [ ] LeanFile
    • [ ] rados
    • [x] us3: https://github.com/beyondstorage/go-service-us3
    • [x] bos: https://github.com/beyondstorage/go-service-bos
    • [x] Huawei Object Storage Service: https://github.com/beyondstorage/go-service-obs
    • [ ] LeanObject
    • [ ] Swift

    Usability and stability improve

    • [x] https://github.com/beyondstorage/go-storage/issues/680
    • [x] https://github.com/beyondstorage/go-storage/issues/646
    • [ ] New connection string layout
    • [ ] go-endpoint support default protocol?

    Related Sprints

    • https://github.com/beyondstorage/community/issues/48
  • ci(*): Upgrade minimum version to Go 1.16

    ci(*): Upgrade minimum version to Go 1.16

    1. Drop Go 1.15 and add Go 1.17 to GitHub Actions workflows. ~2. Remove tidy as the dependency of build target and add another make tidy-all step to build-test.yml. This is required otherwise make build-all will report an error when the current go.mod file indicates a higher Go version than the current installed version.~
    2. The io/ioutil package has been deprecated in Go 1.16 (See https://golang.org/doc/go1.16#ioutil). This PR replaces the existing io/ioutil functions with their new definitions in io and os packages.

    EDIT: Revert Makefile changes

  • feat: Use gg to generate code for pairs

    feat: Use gg to generate code for pairs

    This is a demo PR for evaluations.

    Related to https://github.com/beyondstorage/go-storage/issues/607.

    • Is go-codegen more friendly than go template?
    • Is go-codegen easier to learn/use/understanding?
  • Tracking issue of GSP-970: Service Factory

    Tracking issue of GSP-970: Service Factory

    • GSP: https://github.com/beyondstorage/go-storage/pull/970

    Steps

    • [x] https://github.com/beyondstorage/go-storage/pull/976
    • [ ] Generate factory in serivces
  • Error uploading path with special characters

    Error uploading path with special characters

    s, err := s3.NewStorager(
    	pairs.WithEndpoint("http:127.0.0.1:9000"),
    	pairs.WithCredential("hmac:minio:minio123456"),
    	pairs.WithLocation("us-east-1"),
    	pairs.WithName("/default"),
    	pairs.WithWorkDir("/"),
    )
    
    if err != nil {
    	fmt.Println(err)
    }
    
    f, _ := os.Open("IMG@@@_0960.JPG")
    fi, _ := f.Stat()
    
    fmt.Println(s.WriteWithContext(context.Background(), "IMG@@@¥&_0960.JPG", f, fi.Size())) // error
    fmt.Println(s.WriteWithContext(context.Background(), "IMG_0960.JPG", f, fi.Size()))      // success
    
  • GCS如何配置config string?

    GCS如何配置config string?

    懵逼中,尝试了很多次都失败,貌似是仓库地址和 x-goog-project-id 的问题,能否帮我看下,我该怎么配置连接字符串, 例如:

    access_key: abc
    secret_key: qwe project-id: 123456 url: https://console.cloud.google.com/storage/browser/bucket123 怎么配是对的? gcs://static:abc:qwe@https:console.cloud.google.com:443/bucket123 gcs://static:abc:qwe@https:console.cloud.google.com:443/bucket123/?x-goog-project-id=123456

    都是错的,我该怎么配置?谢谢!

  • services: Unify error handling

    services: Unify error handling

    Proposal: https://github.com/Xuanwo/storage/blob/master/docs/design/11-error-handling.md Spec: https://github.com/Xuanwo/storage/blob/master/docs/spec/1-error-handling.md

    • [x] azblob
    • [x] cos
    • [x] dropbox
    • [x] fs
    • [x] gcs
    • [x] kodo
    • [x] oss
    • [x] qingstor
    • [x] s3
    • [x] uss
  • When migrate to Factory init, make build failed

    When migrate to Factory init, make build failed

    When migrate to Factory init, make build failed with err:

    2021/11/19 10:56:25 not handled type &ast.FuncType{Func:1884, Params:(*ast.FieldList)(0xc0002be0f0), Results:(*ast.FieldList)(0xc0002be120)}
    

    And I just don't know what happend.

  • services/s3: Integration test fails when using the latest MinIO

    services/s3: Integration test fails when using the latest MinIO

    Excepted Behavior

    go-service-s3 should compatible with MinIO and pass integration test with MinIO.

    Current Behavior

    • Integration test passed when using minio RELEASE.2021-06-17T00-10-46Z
    • Decode REST XML response failed when using MinIO Console
      When Read a file     storager.go:55: write: Storager s3 {Name: xxxxx, WorkDir: /dc2c38b3-2de2-413c-8399-92cb1ba879e8/}, [3dd549ed-b614-45b7-9848-d33fdac98238]: unexpected: SerializationError: failed to decode REST XML response
                    status code: 200, request id:
            caused by: XML syntax error on line 1: element <link> closed by </head>
      

    Steps to Reproduce

    Run integration test for go-service-s3 with the following config:

    STORAGE_S3_INTEGRATION_TEST=on
    STORAGE_S3_CREDENTIAL=hmac:minioadmin:minioadmin
    STORAGE_S3_ENDPOINT=https:play.min.io:9443
    STORAGE_S3_NAME=<bucketname>
    STORAGE_S3_LOCATION=local
    

    Also we need to set endpoint for initialization:

    pairs.WithEndpoint("https:play.min.io:9443")
    

    Environment

    • MinIO Console
  • NewStoragerFromString doesn't register service while initiating a new storager

    NewStoragerFromString doesn't register service while initiating a new storager

    Calling NewStoragerFromString using a connection string with the value fs:///tmp, returns ErrServiceNotRegistered.

    I'm using go-service-fs/v3 as the go-storage service.

    I found out that ok's value is returned as false: https://github.com/beyondstorage/go-storage/blob/d1451868532fd29cf90633f65b607eb9ef6e472b/services/new.go#L64-L68

    Although ty is logged as fs in the error below, I think fs is not getting registered as a key in storagerFnMap, due to some issue in RegisterStorager().

    Error Returned:

    2021/07/25 21:08:35 service init failed: fs new_storager: []: service not registered
    exit status 1
    

    Code for reproducing the same error:

    package main
    
    import (
        "bytes"
        "log"
    
        "github.com/beyondstorage/go-storage/v4/pairs"
        "github.com/beyondstorage/go-service-fs/v3"
    )
    
    func main() {
        // Init a service. Fails to the register the service here.
        store, err := services.NewStoragerFromString("fs:///tmp")
        if err != nil {
            log.Fatalf("service init failed: %v", err)
        }
    
        content := []byte("Hello, world!")
        length := int64(len(content))
        r := bytes.NewReader(content)
    
        _, err = store.Write("hello", r, length)
        if err != nil {
            log.Fatalf("write failed: %v", err)
        }
    
        var buf bytes.Buffer
    
        _, err = store.Read("hello", &buf)
        if err != nil {
            log.Fatalf("storager read: %v", err)
        }
    
        log.Printf("%s", buf.String())
    }
    
  • beyondstorage.io website is down causing dependabot to break

    beyondstorage.io website is down causing dependabot to break

    The documentation website and go.beyondstorage.io are both down which causes gitlab-dependabot to throw this error:

    go.beyondstorage.io/services/fs/[email protected]: unrecognized import path "go.beyondstorage.io/services/fs/v4": https fetch: Get "https://go.beyondstorage.io/services/fs/v4?go-get=1": dial tcp: i/o timeout
    

    Dependabot has no way to skip this dependency as ignored dependencies are only skipped after the go mod package manager fetches all new versions. This error also causes dependabot to exit so I have no automated dependency updates for any packages. Can someone please fix the website or not self host it? Great library by the way!

  • fix(services/gcs): Load more then 200 objects from GCS bucket

    fix(services/gcs): Load more then 200 objects from GCS bucket

    When listing bucket that contains > 200 objects, Iterator gets stuck (returns same first 200 objects). Fixed by adding object offset supported by official gcs library.

  • services/obs: Obs does not support read with size and offset

    services/obs: Obs does not support read with size and offset

    Currently obs does not support read with size and offset. When I migrate to Factory init, the integration test requires that TestReadWithOffset item status should be pass. Before I migrate issue #1022, which I should choose?

    1. Ignore read operation
    2. Hang up the migration of obs
Zadig is a cloud native, distributed, developer-oriented continuous delivery product.

Zadig Developer-oriented Continuous Delivery Product English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? Ho

Jan 8, 2023
Zadig is a cloud native, distributed, developer-oriented continuous delivery product.

Zadig Developer-oriented Continuous Delivery Product ⁣ English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use?

May 12, 2021
Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using HPE Smart Storage Administrator tool

hpessa-exporter Overview Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using

Jan 17, 2022
KubeOrbit is an open-source abstraction layer library that turns easy apps testing&debuging on Kubernetes in a new way
KubeOrbit is an open-source abstraction layer library that turns easy apps testing&debuging on Kubernetes in a new way

KubeOrbit is an open-source abstraction layer library that turns easy apps testing&debuging on Kubernetes in a new way

Jan 6, 2023
Linux provisioning scripts + application deployment tools. Suitable for self-hosting and hobby-scale application deployments.

Apollo Linux provisioning scripts + application deployment tools. Suitable for self-hosting and hobby-scale application deployments. Philosophy Linux-

Feb 7, 2022
Flash-metrics - Flash Metrics Storage With Golang

Flash Metrics Storage bootstrap: $ echo -e "max-index-length = 12288" > tidb.con

Jan 8, 2022
Imaginarium - A simple golang image storage engine

Imaginarium A simple golang image storage engine. Used to create and store diffe

Jan 10, 2022
Mini file storage with Go (Golang)

#Microstorage This is my exercise of creating simple file storage with GoLang Purpose: store and manipulate with user`s images in my pet projects ##Ve

Nov 2, 2022
《蓝眼云盘》(Eyeblue Cloud Storage)
《蓝眼云盘》(Eyeblue Cloud Storage)

English Version 蓝眼云盘(3.1.2) 在线Demo (体验账号: demo 密码:123456) 蓝眼云盘是蓝眼开源系列代表作品之一,致力于打造精致,优雅,简约的云盘。核心功能如下: 文件列表 文件/文件夹上传 + 拖拽上传 文件分享 文件监控统计 回收站 多用户 WebDav 扫

Jan 7, 2023
Sample Driver that provides reference implementation for Container Object Storage Interface (COSI) API

cosi-driver-minio Sample Driver that provides reference implementation for Container Object Storage Interface (COSI) API Community, discussion, contri

Oct 10, 2022
topolvm operator provide kubernetes local storage which is light weight and high performance

Topolvm-Operator Topolvm-Operator is an open source cloud-native local storage orchestrator for Kubernetes, which bases on topolvm. Supported environm

Nov 24, 2022
An high performance and ops-free local storage solution for Kubernetes.
An high performance and ops-free local storage solution for Kubernetes.

Carina carina 是一个CSI插件,在Kubernetes集群中提供本地存储持久卷 项目状态:开发测试中 CSI Version: 1.3.0 Carina architecture 支持的环境 Kubernetes:1.20 1.19 1.18 Node OS:Linux Filesys

May 18, 2022
Carina: an high performance and ops-free local storage for kubernetes
Carina: an high performance and ops-free local storage for kubernetes

Carina English | 中文 Background Storage systems are complex! There are more and more kubernetes native storage systems nowadays and stateful applicatio

Dec 30, 2022
Container Storage Interface components for SPIFFE

SPIFFE CSI Driver WARNING: This project is in the "Development" phase of the SPIFFE Project Maturity Phases. A Container Storage Interface driver for

Jan 3, 2023
StaticBackend is a simple backend server API handling user mgmt, database, storage and real-time component
StaticBackend is a simple backend server API handling user mgmt, database, storage and real-time component

StaticBackend is a simple backend that handles user management, database, file storage, forms, and real-time experiences via channel/topic-based communication for web and mobile applications.

Jan 7, 2023
A set of components that can be composed into a highly available metric system with unlimited storage capacity
A set of components that can be composed into a highly available metric system with unlimited storage capacity

Overview Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added

Oct 20, 2021
Dynamically provisioning persistent local storage with Kubernetes

Local Path Provisioner Overview Local Path Provisioner provides a way for the Kubernetes users to utilize the local storage in each node. Based on the

Jan 4, 2023
A letsencrypt client that uses etcd as its storage.

letsencrypt-with-etcd This is a letsencrypt client that uses etcd as its storage. It stores your (automatically created) LetsEncrypt account in /letse

Jan 20, 2022
SMART information of local storage devices as Prometheus metrics

hpessa-exporter Overview Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using

Feb 10, 2022