s3fs provides a S3 implementation for Go1.16 filesystem interface.

s3fs Go Reference Go

Package s3fs provides a S3 implementation for Go1.16 filesystem interface.

Since S3 is a flat structure, s3fs simulates directories by using prefixes and "/" delim. ModTime on directories is always zero value.

The implementation wraps aws sdk go s3 client.

const bucket = "my-bucket"

s, err := session.NewSession()
if err != nil {
    log.Fatal(err)
}

s3fs := s3fs.New(s3.New(s), bucket)

// print out all files in s3 bucket.
_ = fs.WalkDir(s3fs, ".", func(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err
    }

    if d.IsDir() {
        fmt.Println("dir:", path)
        return nil
    }
    fmt.Println("file:", path)
    return nil
})

Installation

go get github.com/jszwec/s3fs

Requirements

  • Go1.16+
Owner
Similar Resources

A Go filesystem package for working with files and directories

Stowage A Go filesystem package for working with files and directories, it features a simple API with support for the common files and directories ope

May 28, 2021

filesystem for golang

filesystem filesystem for golang installation go get github.com/go-component/filesystem import import "github.com/go-component/filesystem" Usage sup

Nov 1, 2022

A set of io/fs filesystem abstractions and utilities for Go

A set of io/fs filesystem abstractions and utilities for Go

A set of io/fs filesystem abstractions and utilities for Go Please ⭐ this project Overview This package provides io/fs interfaces for: Cloud providers

Nov 19, 2022

Tarserv serves streaming tar files from filesystem snapshots.

tarserv A collection of tools that allow serving large datasets from local filesystem snapshots. It is meant for serving big amounts of data to shell

Jan 11, 2022

Grep archive search in any files on the filesystem, in archive and even inner archive.

grep-archive Grep archive search for string in any files on the filesystem, in archive and even inner archive. Supported archive format are : Tar Form

Jan 26, 2022

Warp across your filesystem in ~5 ms

Warp across your filesystem in ~5 ms

WarpDrive: the Go version. What does this do? Instead of having a huge cd routine to get where you want, with WarpDrive you use short keywords to warp

Dec 14, 2022

A go compress library for fs.FS interface

compress: a go compress library for fs.FS interface Format Test Charset Decoder Encoder Password Info zip local true true true false used go std rar l

Aug 1, 2022

Go implementation of the Open Packaging Conventions (OPC)

opc Package opc implements the ISO/IEC 29500-2, also known as the Open Packaging Convention. The Open Packaging specification describes an abstract mo

Oct 14, 2022

A test repo to demonstrate the current (go1.17.2) issue when trying to use retractA test repo to demonstrate the current (go1.17.2) issue when trying to use retract

test-go-mod-retract This is a test repo to demonstrate the current (go1.17.2) issue when trying to use retract in go.mod to retract a version in a non

Oct 16, 2021

An implementation of the FileSystem interface for tar files.

TarFS A wrapper around tar.Reader. Implements the FileSystem interface for tar files. Adds an Open method, that enables reading of file according to i

Sep 26, 2022

Package gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends.

sessions gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends. The key features are: Simple API: us

Dec 28, 2022

Some helper types for go1: priority queue, slice wrapper.

go-villa Package villa contains some helper types for Go: priority queue, slice wrapper, binary-search, merge-sort. GoDoc Link: http://godoc.org/githu

Apr 24, 2021

Read a tar file contents using go1.16 io/fs abstraction

Read a tar file contents using go1.16 io/fs abstraction

go-tarfs Read a tar file contents using go1.16 io/fs abstraction Usage ⚠️ go-tarfs needs go=1.16 Install: go get github.com/nlepage/go-tarfs Use: pac

Dec 1, 2022

YSHOP-GO基于当前流行技术组合的前后端RBAC管理系统:Go1.15.x+Beego2.x+Jwt+Redis+Mysql8+Vue 的前后端分离系统,权限控制采用 RBAC,支持数据字典与数据权限管理,支持动态路由等

YSHOP-GO 后台管理系统 项目简介 YSHOP-GO基于当前流行技术组合的前后端RBAC管理系统:Go1.15.x+Beego2.x+Jwt+Redis+Mysql8+Vue 的前后端分离系统,权限控制采用 RBAC,支持数据字典与数据权限管理,支持动态路由等 体验地址: https://go

Dec 30, 2022

beego-vue-admin基于当前流行技术组合的前后端RBAC管理系统:Go1.15.x+Beego2.x+Jwt+Redis+Mysql8+Vue 的前后端分离系统,权限控制采用 RBAC,支持数据字典与数据权限管理,支持动态路由等

beego-vue-admin基于当前流行技术组合的前后端RBAC管理系统:Go1.15.x+Beego2.x+Jwt+Redis+Mysql8+Vue 的前后端分离系统,权限控制采用 RBAC,支持数据字典与数据权限管理,支持动态路由等

beego-vue-admin 后台管理系统 项目简介 beego-vue-admin基于当前流行技术组合的前后端RBAC管理系统:Go1.15.x+Beego2.x+Jwt+Redis+Mysql8+Vue 的前后端分离系统,权限控制采用 RBAC,支持数据字典与数据权限管理,支持动态路由等 体验

Dec 30, 2022

experimental promises in go1.18 with generics

async go a prototype of "promises" in go1.18. note: this is just an experiment used to test alternate patterns for dealing with asynchronous code in g

Dec 23, 2022

A go1.18 wrapper to provide simple generics based API for defining command line flags.

gflag A go1.18 wrapper to provide simple generics based API for defining command line flags. Example package main import ( "flag" "fmt" "time" "

Dec 20, 2021

A go1.18+ package to (maybe) simplify performing operations on slices in a fluent-like style.

sop ✨ W.I.P. ✨ sop (slices operation) is a go1.18+ package to (maybe) simplify performing operations on slices in a fluent-like style with common oper

Oct 1, 2022

Go.work-workspace-example - Go1.18 workspace example

Go.work-workspace-example - Go1.18 workspace example

Jan 20, 2022
Comments
  • Unsafe functional options

    Unsafe functional options

    I noticed you use functional options (which is great), but way they are currently implemented is not perfectly concurrent safe.

    Here is the option type:

    // Option is a function that provides optional features to S3FS.
    type Option func(*S3FS)
    

    The problem with this signature is that you can use options after initialization to modify the state of S3FS:

    WithReadSeeker(New(...))
    

    You can avoid that by introducing a private option struct:

    // Option is a function that provides optional features to S3FS.
    type Option func(*s3FSoptions)
    

    An even better solution is making the Option type an interface:

    // Option is a function that provides optional features to S3FS.
    type Option interface{
        apply(*S3FS)
    }
    

    Using any of the above methods callers won't be able to modify internal state after initialization.

  • stat fails for existing directory

    stat fails for existing directory

    I believe I've identified a bug that causes Stat() to fail in certain circumstances. Given two objects, a/b.txt and a.txt, a call to fs.Stat(fsys, "a") will fail: open a: file does not exist. I believe this is because the MaxKeys budget (of 1) in the call to ListObjects is used-up by a.txt in the response Contents (link).

    I'll submit a failing test in a PR

  • Switch to v2 of the aws sdk

    Switch to v2 of the aws sdk

    Amazon has a version 2 of the api which is already the official predecessor of v1.

    The main advantages of the new API are performance improvements, easier error handling, easier credentials handling, ...

    https://aws.amazon.com/blogs/developer/client-updates-in-the-preview-version-of-the-aws-sdk-for-go-v2/

Read a tar file contents using go1.16 io/fs abstraction
Read a tar file contents using go1.16 io/fs abstraction

go-tarfs Read a tar file contents using go1.16 io/fs abstraction Usage ⚠️ go-tarfs needs go>=1.16 Install: go get github.com/nlepage/go-tarfs Use: pac

Dec 1, 2022
A Go io/fs filesystem implementation for reading files in a Github gists.

GistFS GistFS is an io/fs implementation that enables to read files stored in a given Gist. Requirements This module depends on io/fs which is only av

Oct 14, 2022
A FileSystem Abstraction System for Go
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Jan 9, 2023
A package to allow one to concurrently go through a filesystem with ease

skywalker Skywalker is a package to allow one to concurrently go through a filesystem with ease. Features Concurrency BlackList filtering WhiteList fi

Nov 14, 2022
Takes an input http.FileSystem (likely at go generate time) and generates Go code that statically implements it.

vfsgen Package vfsgen takes an http.FileSystem (likely at go generate time) and generates Go code that statically implements the provided http.FileSys

Dec 18, 2022
memfs: A simple in-memory io/fs.FS filesystem

memfs: A simple in-memory io/fs.FS filesystem memfs is an in-memory implementation of Go's io/fs.FS interface. The goal is to make it easy and quick t

Jan 8, 2023
A Small Virtual Filesystem in Go

This is a virtual filesystem I'm coding to teach myself Go in a fun way. I'm documenting it with a collection of Medium posts that you can find here.

Dec 11, 2022
CRFS: Container Registry Filesystem

CRFS: Container Registry Filesystem Discussion: https://github.com/golang/go/issues/30829 Overview CRFS is a read-only FUSE filesystem that lets you m

Dec 26, 2022
Encrypted overlay filesystem written in Go
Encrypted overlay filesystem written in Go

An encrypted overlay filesystem written in Go. Official website: https://nuetzlich.net/gocryptfs (markdown source). gocryptfs is built on top the exce

Jan 8, 2023
Go filesystem implementations for various URL schemes

hairyhenderson/go-fsimpl This module contains a collection of Go filesystem implementations that can discovered dynamically by URL scheme. All filesys

Dec 28, 2022