Abstract File Storage

afs - abstract file storage

GoReportCard GoDoc goversion-image gopherbadger-tag-do-not-edit

Please refer to CHANGELOG.md if you encounter breaking changes.

Motivation

When dealing with various storage systems, like cloud storage, SCP, container or local file system, using shared API for typical storage operation provides an excellent simplification. What's more, the ability to simulate storage-related errors like Auth or EOF allows you to test an app error handling.

Introduction

This library uses a storage manager abstraction to provide an implementation for a specific storage system with following

  • CRUD Operation:
List(ctx context.Context, URL string, options ...Option) ([]Object, error)

Walk(ctx context.Context, URL string, handler OnVisit, options ...Option) error

Open(ctx context.Context, object Object, options ...Option) (io.ReadCloser, error)

OpenURL(ctx context.Context, URL string, options ...Option) (io.ReadCloser, error)


Upload(ctx context.Context, URL string, mode os.FileMode, reader io.Reader, options ...Option) error

Create(ctx context.Context, URL string, mode os.FileMode, isDir bool, options ...Option) error

Delete(ctx context.Context, URL string, options ...Option) error
  • Batch uploader:
type Upload func(ctx context.Context, parent string, info os.FileInfo, reader io.Reader) error
 
Uploader(ctx context.Context, URL string, options ...Option) (Upload, io.Closer, error)
  • Utilities:
Copy(ctx context.Context, sourceURL, destURL string, options ...Option) error

Move(ctx context.Context, sourceURL, destURL string, options ...Option) error

NewWriter(ctx context.Context, URL string, mode os.FileMode, options ...storage.Option) (io.WriteCloser, error)

DownloadWithURL(ctx context.Context, URL string, options ...Option) ([]byte, error)

Download(ctx context.Context, object Object, options ...Option) ([]byte, error)

URL scheme is used to identify storage system, or alternatively relative/absolute path can be used for local file storage. By default, all operations using the same baseURL share the same corresponding storage manager instance. For example, instead supplying SCP auth details for all operations, auth option can be used only once.

func main() {

    ctx := context.Background()
    {
        //auth with first call 
        fs := afs.New()
        defer fs.Close()
        keyAuth, err := scp.LocalhostKeyAuth("")
        if err != nil {
           log.Fatal(err)
        }
        reader1, err := fs.OpenURL(ctx, "scp://host1:22/myfolder/asset.txt", keyAuth)
        if err != nil {
               log.Fatal(err)
        }
        ...
        reader2, err := fs.OpenURL(ctx, "scp://host1:22/myfolder/asset.txt", keyAuth)
    }
    
    {
        //auth per baseURL 
        fs := afs.New()
        err = fs.Init(ctx, "scp://host1:22/", keyAuth)
        if err != nil {
            log.Fatal(err)
        }
        defer fs.Destroy("scp://host1:22/")
        reader, err := fs.OpenURL(ctx, "scp://host1:22/myfolder/asset.txt")
     }
}

Usage

Downloading location content
func main() {
	
    fs := afs.New()
    ctx := context.Background()
    objects, err := fs.List(ctx, "/tmp/folder")
    if err != nil {
        log.Fatal(err)
    }
    for _, object := range objects {
        fmt.Printf("%v %v\n", object.Name(), object.URL())
        if object.IsDir() {
            continue
        }
        reader, err := fs.Open(ctx, object)
        if err != nil {
            log.Fatal(err)
        }
        data, err := ioutil.ReadAll(reader)
        if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%s\n", data)
    }
}
Uploading Content
func main() {
	
    fs := afs.New()
    ctx := context.Background()
    keyAuth, err := scp.LocalhostKeyAuth("")
    if err != nil {
        log.Fatal(err)
    }
    err  = fs.Init(ctx, "scp://127.0.0.1:22/", keyAuth)
    if err != nil {
        log.Fatal(err)
    }	
    err = fs.Upload(ctx, "scp://127.0.0.1:22/folder/asset.txt", 0644, strings.NewReader("test me"))
    if err != nil {
        log.Fatal(err)
    }
    ok, err := fs.Exists(ctx, "scp://127.0.0.1:22/folder/asset.txt")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("has file: %v\n", ok)
    _ = fs.Delete(ctx, "scp://127.0.0.1:22/folder/asset.txt")
}
Uploading Content With Writer
func main() {
	
    fs := afs.New()
    ctx := context.Background()
    keyAuth, err := scp.LocalhostKeyAuth("")
    if err != nil {
        log.Fatal(err)
    }
    err  = fs.Init(ctx, "scp://127.0.0.1:22/", keyAuth)
    if err != nil {
        log.Fatal(err)
    }	
    writer = fs.NewWriter(ctx, "scp://127.0.0.1:22/folder/asset.txt", 0644)
    _, err := writer.Write([]byte("test me")))
    if err != nil {
        log.Fatal(err)
    }
    err = writer.Close()
    if err != nil {
        log.Fatal(err)
    }
    ok, err := fs.Exists(ctx, "scp://127.0.0.1:22/folder/asset.txt")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("has file: %v\n", ok)
    _ = fs.Delete(ctx, "scp://127.0.0.1:22/folder/asset.txt")
}
Data Copy
func main() {

    fs := afs.New()
    ctx := context.Background()
    keyAuth, err := scp.LocalhostKeyAuth("")
    if err != nil {
        log.Fatal(err)
    }
    err = fs.Copy(ctx, "s3://mybucket/myfolder", "scp://127.0.0.1/tmp", option.NewSource(), option.NewDest(keyAuth))
    if err != nil {
        log.Fatal(err)
    }
}
Archiving content
func main() {
	
    secretPath := path.Join(os.Getenv("HOME"), ".secret", "gcp-e2e.json")
    auth, err := gs.NewJwtConfig(option.NewLocation(secretPath))
    if err != nil {
        return
    }
    sourceURL := "mylocalPath/"
    destURL := "gs:mybucket/test.zip/zip://localhost/dir1"
    fs := afs.New()
    ctx := context.Background()
    err = fs.Copy(ctx, sourceURL, destURL, option.NewDest(auth))
    if err != nil {
        log.Fatal(err)
    }

}	
Archive Walker

Walker can be created for tar or zip archive.

func main() {
	
    ctx := context.Background()
	fs := afs.New()
	walker := tar.NewWalker(s3afs.New())
	err := fs.Copy(ctx, "/tmp/test.tar", "s3:///dest/folder/test", walker)
	if err != nil {
		log.Fatal(err)
	}
Archive Uploader

Uploader can be created for tar or zip archive.

func main() {
	
    ctx := context.Background()
	fs := afs.New()
	uploader := zip.NewBatchUploader(gsafs.New())
	err := fs.Copy(ctx, "gs:///tmp/test/data", "/tmp/data.zip", uploader)
	if err != nil {
		log.Fatal(err)
	}
}
Data Move
func main() {
	
    fs := afs.New()
	ctx := context.Background()
	keyAuth, err := scp.LocalhostKeyAuth("")
	if err != nil {
		log.Fatal(err)
	}
	err = fs.Move(ctx, "/tmp/transient/app", "scp://127.0.0.1/tmp", option.NewSource(), option.NewDest(keyAuth))
	if err != nil {
		log.Fatal(err)
	}
}	
Batch Upload
func main() {
	
    fs := afs.New()
	ctx := context.Background()
	upload, closer, err := fs.Uploader(ctx, "/tmp/clone")
	if err != nil {
		log.Fatal(err)
	}
	defer closer.Close()
	assets := []*asset.Resource{
		asset.NewFile("asset1.txt", []byte("test 1"), 0644),
		asset.NewFile("asset2.txt", []byte("test 2"), 0644),
		asset.NewDir("folder1", file.DefaultDirOsMode),
		asset.NewFile("folder1/asset1.txt", []byte("test 3"), 0644),
		asset.NewFile("folder1/asset2.txt", []byte("test 4"), 0644),
	}
	for _, asset := range assets {
		relative := ""
		var reader io.Reader
		if strings.Contains(asset.Name, "/") {
			relative, _ = path.Split(asset.Name)
		}
		if ! asset.Dir {
			reader = bytes.NewReader(asset.Data)
		}
		err = upload(ctx, relative, asset.Info(), reader)
		if err != nil {
			log.Fatal(err)
		}
	}
}

Matchers

To filter source content you can use Matcher option. The following have been implemented.

Basic Matcher

func main() {
	
    matcher, err := NewBasic("/data", ".avro", nil)
    fs := afs.New()
    ctx := context.Background()
    err := fs.Copy(ctx, "/tmp/data", "s3://mybucket/data/", matcher.Match)
    if err != nil {
        log.Fatal(err)
    }
}

Exclusion

func main() {
	
    matcher := matcher.Basic{Exclusion:".+/data/perf/\\d+/.+"}
    fs := afs.New()
    ctx := context.Background()
    err := fs.Copy(ctx, "/tmp/data", "s3://mybucket/data/", matcher.Match)
    if err != nil {
        log.Fatal(err)
    }
}

Filepath matcher

OS style filepath match, with the following terms:

  • '*' matches any sequence of non-Separator characters
  • '?' matches any single non-Separator character
  • '[' [ '^' ] { character-range } ']'
func main() {
	
    matcher := matcher.Filepath("*.avro")
    fs := afs.New()
    ctx := context.Background()
    err := fs.Copy(ctx, "/tmp/data", "gs://mybucket/data/", matcher)
    if err != nil {
        log.Fatal(err)
    }
}	
		

Ignore Matcher

Ignore matcher represents matcher that matches file that are not in the ignore rules. The syntax of ignore borrows heavily from that of .gitignore; see https://git-scm.com/docs/gitignore or man gitignore for a full reference.

func mian(){
	
	ignoreMatcher, err := matcher.NewIgnore([]string{"*.txt", ".ignore"})
  	//or matcher.NewIgnore(option.NewLocation(".cloudignore"))
	if err != nil {
		log.Fatal(err)
	}
	fs := afs.New()
	ctx := context.Background()
	objects, err := fs.List(ctx, "/tmp/folder", ignoreMatcher.Match)
	if err != nil {
		log.Fatal(err)
	}
	for _, object := range objects {
		fmt.Printf("%v %v\n", object.Name(), object.URL())
		if object.IsDir() {
			continue
		}
	}
}	

Modification Time Matcher

Modification Time Matcher represents matcher that matches file that were modified either before or after specified time.

func mian(){
	
	before, err := toolbox.TimeAt("2 days ago in UTC")
    if err != nil {
		log.Fatal(err)
	}	
	modTimeMatcher, err := matcher.NewModification(before, nil)
	if err != nil {
		log.Fatal(err)
	}
	fs := afs.New()
	ctx := context.Background()
	objects, err := fs.List(ctx, "/tmp/folder", modTimeMatcher.Match)
	if err != nil {
		log.Fatal(err)
	}
	for _, object := range objects {
		fmt.Printf("%v %v\n", object.Name(), object.URL())
		if object.IsDir() {
			continue
		}
	}
}	

Content modifiers

To modify resource content on the fly you can use Modifier option.

func main() {
	fs := afs.New()
	ctx := context.Background()
	sourceURL := "file:/tmp/app.war/zip://localhost/WEB-INF/classes/config.properties"
	destURL := "file:/tmp/app.war/zip://localhost/"
	err := fs.Copy(ctx, sourceURL, destURL, modifier.Replace(map[string]string{
		"${changeMe}": os.Getenv("USER"),
	}))
	if err != nil {
		log.Fatal(err)
	}
}
package main

import (
	"context"
	"log"
	"github.com/viant/afs"
	"io"
	"fmt"
	"io/ioutil"
	"os"
	"strings"
)

func modifyContent(info os.FileInfo, reader io.ReadCloser) (closer io.ReadCloser, e error) {
   if strings.HasSuffix(info.Name() ,".info") {
       data, err := ioutil.ReadAll(reader)
       if err != nil {
           return nil, err
       }
       _ = reader.Close()
       expanded := strings.Replace(string(data), "$os.User", os.Getenv("USER"), 1)
       reader = ioutil.NopCloser(strings.NewReader(expanded))
   }
   return reader, nil
}                           

func main() {

    fs := afs.New()
    reader ,err := fs.OpenURL(context.Background(), "s3://mybucket/meta.info", modifyContent)
    if err != nil {
        log.Fatal(err)	
    }
    
    defer reader.Close()
    content, err := ioutil.ReadAll(reader)
    if err != nil {
        log.Fatal(err)	
    }
    fmt.Printf("content: %s\n", content)
	
}

Streaming data

Streaming data allows data reading and uploading in chunks with small memory footprint.

    jwtConfig, err := gs.NewJwtConfig()
	if err != nil {
		log.Fatal(err)
	}

	ctx := context.Background()
	fs := afs.New()
	sourceURL := "gs://myBucket/path/myasset.gz"
	reader, err := fs.OpenURL(ctx, sourceURL, jwtConfig, option.NewStream(64*1024*1024, 0))
	if err != nil {
		log.Fatal(err)
	}
    
	_ = os.Setenv("AWS_SDK_LOAD_CONFIG", "true")
	destURL := "s3://myBucket/path/myasset.gz"
	err = fs.Upload(ctx, destURL, 0644, reader, &option.Checksum{Skip:true})
	if err != nil {
		log.Fatal(err)
	}

    // or
    writer = fs.NewWriter(ctx, destURL, 0644, &option.Checksum{Skip:true})
    _, err = io.Copy(writer, reader)
    if err != nil {
        log.Fatal(err)
    }
    err = writer.Close()
	if err != nil {
		log.Fatal(err)
	}

Options

To control number and position of listed resources you can yse page option.

Provider specific timeout.

Provides user/password auth.

  • Source & Dest Options

Groups options by source or destination options. This options work with Copy or Move operations.

func main() {
	
    fs := afs.New()
    secretPath :=  path.Join(os.Getenv("HOME"), ".secret","gcp.json")
    jwtConfig, err := gs.NewJwtConfig(option.NewLocation(secretPath))
    if err != nil {
    	log.Fatal(err)
    }
    sourceOptions := option.NewSource(jwtConfig)
    authConfig, err := s3.NewAuthConfig(option.NewLocation("aws.json"))
    if err != nil {
        log.Fatal(err)
    }
    destOptions := option.NewDest(authConfig)
	err = fs.Copy(ctx, "gs://mybucket/data", "s3://mybucket/data",  sourceOptions, destOptions)
}
  • option.Checksum skip computing checksum if Skip is set, this option allows streaming upload in chunks
  • option.Stream: download reader reads data with specified stream PartSize

Check out storage manager for additional options.

Storage Implementations

Testing fs

To unit test all storage operation all in memory you can use faker fs.

In addition you can use error options to test exception handling.

  • DownloadError
func mian() {
	fs := afs.NewFaker()
	ctx := context.Background()
	err := fs.Upload(ctx, "gs://myBucket/folder/asset.txt", 0, strings.NewReader("some data"), option.NewUploadError(io.EOF))
	if err != nil {
		log.Fatalf("expect upload error: %v", err)
	}
}
  • ReaderError
func mian() {
    fs := afs.NewFaker()
	ctx := context.Background()
	err := fs.Upload(ctx, "gs://myBucket/folder/asset.txt", 0, strings.NewReader("some data"), option.NewDownloadError(io.EOF))
	if err != nil {
		log.Fatal(err)
	}
	_, err = fs.OpenURL(ctx, "gs://myBucket/folder/asset.txt")
	if err != nil {
		log.Fatalf("expect download error: %v", err)
	}
}
  • UploadError
func mian() {
    fs := afs.NewFaker()
    ctx := context.Background()
    err := fs.Upload(ctx, "gs://myBucket/folder/asset.txt", 0, strings.NewReader("some data"), option.NewUploadError(io.EOF))
    if err != nil {
        log.Fatalf("expect upload error: %v", err)
    }
}
Code generation for static or in memory go file

Generate with mem storage

package main

import (
    "log"
    "github.com/viant/afs/parrot
)

func mian() {
  ctx := context.Background()
  err := parrot.GenerateWithMem(ctx, "pathToBinaryAsset", "gen.go", false)
  if err != nil {
    log.Fatal(err)
  }
}

Generate static data files

package main

import (
    "log"
    "github.com/viant/afs/parrot
)

func mian() {
  ctx := context.Background()
  err := parrot.Generate(ctx, "pathToBinaryAsset", "data/", false)
  if err != nil {
    log.Fatal(err)
  }
}

Test setup utilities

Package asset defines basic utilities to quickly manage asset related unit tests.

func Test_XXX(t *testing.T) {

    var useCases = []struct {
		description string
		location    string
		options     []storage.Option
		assets      []*asset.Resource
	}{

	}

	ctx := context.Background()
	for _, useCase := range useCases {
		fs := afs.New()
		mgr, err := afs.Manager(useCase.location, useCase.options...)
		if err != nil {
			log.Fatal(err)
		}
		err = asset.Create(mgr, useCase.location, useCase.assets)
		if err != nil {
			log.Fatal(err)
		}
		
		//... actual app logic

		actuals, err := asset.Load(mgr, useCase.location)
		if err != nil {
			log.Fatal(err)
		}
        for _, expect := range useCase.assets {
            actual, ok := actuals[expect.Name]
            if !assert.True(t, ok, useCase.description+": "+expect.Name+fmt.Sprintf(" - actuals: %v", actuals)) {
                continue
            }
            assert.EqualValues(t, expect.Name, actual.Name, useCase.description+" "+expect.Name)
            assert.EqualValues(t, expect.Mode, actual.Mode, useCase.description+" "+expect.Name)
            assert.EqualValues(t, expect.Dir, actual.Dir, useCase.description+" "+expect.Name)
            assert.EqualValues(t, expect.Data, actual.Data, useCase.description+" "+expect.Name)
        }

		_ = asset.Cleanup(mgr, useCase.location)

	}
}

GoCover

GoCover

License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE.

Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.

Credits and Acknowledgements

Library Author: Adrian Witas

Owner
Comments
  • How to specify remote scp username

    How to specify remote scp username

    Hello,

    I was wondering how to specify the username of the remote user when using scp with LocalhostKeyAuth. It appears to try and connect as the user that is running the program (USER from env). I tried modifying the url to scp://user@host:22/path/to/file but that does not work.

    Thanks!

    -Will

  • Difference from affero

    Difference from affero

    Hello, thanks for great lib.

    If possible, can you briefly specify, what is afs different from https://github.com/spf13/afero in it's motivation and implementation.

    And can we pass opened file to other tools which expects native file interface?

    Thanks.

  • Set content type as option

    Set content type as option

    Hi!

    I want to set the content type of the file when i save it to cloud storage (ex: image/jpeg), but cant find the correct option for my call:

    wc, err := afs.New().NewWriter(ctx, dest, 0644)
    

    (by the way thanks for this package, its awesome!)

  • Pass `ssh.ClientConfig` to scp.Service

    Pass `ssh.ClientConfig` to scp.Service

    Currently I do not see a way to pass a ssh.ClientConfig to the scp.Service. The readme on https://github.com/viant/afs/tree/master/scp shows an example for the scp.Storage to do so.

    Reason for this is the default MACs set by Go for SSH are not compatible with the one's we currently expose on servers.

    If this is not something currently possible, I can send a PR that adds the functionality, but I am unsure if I am maybe missing something obvious.

  • Build error:  ../../vendor/github.com/viant/afs/embed/manager.go:5:2: cannot find package

    Build error: ../../vendor/github.com/viant/afs/embed/manager.go:5:2: cannot find package "embed" in any of

    I am getting this build error:

    ../../vendor/github.com/viant/afs/embed/manager.go:5:2: cannot find package "embed" in any of:
            /home/Murph/go/src/github.com/bluearchive/main/vendor/embed (vendor tree)
            /usr/local/go/src/embed (from $GOROOT)
            /home/Murph/go/src/embed (from $GOPATH)
    

    I am not trying to use the embed package, but it appears that what I am trying to use requires it. Here are my imports:

            "github.com/viant/afs"
            "github.com/viant/afs/scp"
            "github.com/viant/afs/option"
    

    And here is all the code I have written to sfar:

    auth := scp.NewAuthProvider (nil, option.NewBasicAuth ("user", "password"))
    service := afs.New()
    

    I ran dep ensure, which put the packages into my vendor/github.com/viant/afs directory. It also put the embed package there.

    The start of afs/embed/manager.go looks like this:

    package embed
    
    import (
            "context"
            "embed"
    

    This package appears to be trying to import itself, which I have never seen.

    My Go version is 1.12.17.

    Any ideas why this isn't working?

    Thanks.

  • Panic during failing TestNewFaker test. Invalid memory address or nil pointer dereference

    Panic during failing TestNewFaker test. Invalid memory address or nil pointer dereference

    Hi. Tried to run some tests - but caught this runtime error:

    $ go test -v -x WORK=/tmp/go-build367362019 mkdir -p $WORK/b001/ mkdir -p $WORK/b024/ mkdir -p $WORK/b016/ mkdir -p $WORK/b040/ mkdir -p $WORK/b043/ mkdir -p $WORK/b042/ mkdir -p $WORK/b025/ mkdir -p $WORK/b044/ mkdir -p $WORK/b049/ mkdir -p $WORK/b012/ mkdir -p $WORK/b050/ mkdir -p $WORK/b039/ mkdir -p $WORK/b037/ mkdir -p $WORK/b057/ mkdir -p $WORK/b056/ mkdir -p $WORK/b079/ mkdir -p $WORK/b077/ mkdir -p $WORK/b076/ mkdir -p $WORK/b082/ mkdir -p $WORK/b084/ mkdir -p $WORK/b083/ mkdir -p $WORK/b080/ mkdir -p $WORK/b089/ mkdir -p $WORK/b090/ mkdir -p $WORK/b059/ mkdir -p $WORK/b091/ mkdir -p $WORK/b092/ mkdir -p $WORK/b096/ mkdir -p $WORK/b093/ mkdir -p $WORK/b053/ mkdir -p $WORK/b097/ mkdir -p $WORK/b008/ mkdir -p $WORK/b099/ mkdir -p $WORK/b098/ cat >$WORK/b001/importcfg.link << 'EOF' # internal packagefile /home/oceanfish81/other_golang/afs.test=/home/oceanfish81/.cache/go-build/ff/fff91e1020f228c2c6937deebda67239b8f01b222833b4d42e8193322d1fc2b7-d packagefile /home/oceanfish81/other_golang/afs=/home/oceanfish81/.cache/go-build/49/499cce08ff4968b086994ee7b46063c89fdc0451cda3a81cc6d706602884b519-d packagefile /home/oceanfish81/other_golang/afs_test=/home/oceanfish81/.cache/go-build/c4/c4a5e50e3037587830d2a953c4eaf9e98cbf5d4b454e1ac9111a61f7130cfcb5-d packagefile github.com/stretchr/testify/assert=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/stretchr/testify/libassert.a packagefile github.com/viant/afs/asset=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libasset.a packagefile github.com/viant/afs/file=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libfile.a packagefile github.com/viant/afs/option=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/liboption.a packagefile github.com/viant/afs/scp=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libscp.a packagefile github.com/viant/afs/storage=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libstorage.a packagefile github.com/viant/afs/url=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/liburl.a packagefile github.com/pkg/errors=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/pkg/liberrors.a packagefile github.com/viant/afs/base=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libbase.a packagefile github.com/viant/afs/http=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libhttp.a packagefile github.com/viant/afs/matcher=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libmatcher.a packagefile github.com/viant/afs/mem=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libmem.a packagefile github.com/viant/afs/ssh=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libssh.a packagefile github.com/viant/afs/tar=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libtar.a packagefile github.com/viant/afs/walker=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libwalker.a packagefile github.com/viant/afs/zip=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libzip.a packagefile github.com/viant/afs=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/libafs.a packagefile github.com/davecgh/go-spew/spew=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/davecgh/go-spew/libspew.a packagefile github.com/pmezard/go-difflib/difflib=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/pmezard/go-difflib/libdifflib.a packagefile gopkg.in/yaml.v3=/home/oceanfish81/go/pkg/gccgo_linux_amd64/gopkg.in/libyaml.v3.a packagefile github.com/viant/afs/object=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libobject.a packagefile golang.org/x/crypto/ssh=/home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libssh.a packagefile github.com/go-errors/errors=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/go-errors/liberrors.a packagefile github.com/viant/afs/archive=/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libarchive.a packagefile golang.org/x/crypto/chacha20=/home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libchacha20.a packagefile golang.org/x/crypto/curve25519=/home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libcurve25519.a packagefile golang.org/x/crypto/ed25519=/home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libed25519.a packagefile golang.org/x/crypto/poly1305=/home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libpoly1305.a packagefile golang.org/x/crypto/ssh/internal/bcrypt_pbkdf=/home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/ssh/internal/libbcrypt_pbkdf.a packagefile golang.org/x/crypto/internal/subtle=/home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/internal/libsubtle.a packagefile golang.org/x/crypto/blowfish=/home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libblowfish.a EOF cd . /usr/local/bin/llvm-goc -o $WORK/b001/afs.test "-Wl,-(" -m64 -Wl,--whole-archive /home/oceanfish81/.cache/go-build/ff/fff91e1020f228c2c6937deebda67239b8f01b222833b4d42e8193322d1fc2b7-d /home/oceanfish81/.cache/go-build/49/499cce08ff4968b086994ee7b46063c89fdc0451cda3a81cc6d706602884b519-d /home/oceanfish81/.cache/go-build/c4/c4a5e50e3037587830d2a953c4eaf9e98cbf5d4b454e1ac9111a61f7130cfcb5-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/stretchr/testify/libassert.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libasset.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libfile.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/liboption.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libscp.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libstorage.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/liburl.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/pkg/liberrors.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libbase.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libhttp.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libmatcher.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libmem.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libssh.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libtar.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libwalker.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libzip.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/libafs.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/davecgh/go-spew/libspew.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/pmezard/go-difflib/libdifflib.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/gopkg.in/libyaml.v3.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libobject.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libssh.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/go-errors/liberrors.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libarchive.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libchacha20.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libcurve25519.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libed25519.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libpoly1305.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/ssh/internal/libbcrypt_pbkdf.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/internal/libsubtle.a /home/oceanfish81/go/pkg/gccgo_linux_amd64/golang.org/x/crypto/libblowfish.a -Wl,--no-whole-archive "-Wl,-)" -Wl,--build-id=0x6233466a383071436478784f506d7962474462722f6b485634634d7144524962414a6938514a37546d2f71454b4e354a614a4a6c7473426b5a73544473492f6233466a383071436478784f506d796247446272 cat >$WORK/b008/vet.cfg << 'EOF' # internal { "ID": "/home/oceanfish81/other_golang/afs", "Compiler": "gccgo", "Dir": "/home/oceanfish81/other_golang/afs", "ImportPath": "/home/oceanfish81/other_golang/afs", "GoFiles": [ "/home/oceanfish81/other_golang/afs/copy.go", "/home/oceanfish81/other_golang/afs/doc.go", "/home/oceanfish81/other_golang/afs/faker.go", "/home/oceanfish81/other_golang/afs/init.go", "/home/oceanfish81/other_golang/afs/list.go", "/home/oceanfish81/other_golang/afs/move.go", "/home/oceanfish81/other_golang/afs/registry.go", "/home/oceanfish81/other_golang/afs/service.go", "/home/oceanfish81/other_golang/afs/uploader.go", "/home/oceanfish81/other_golang/afs/walker.go", "/home/oceanfish81/other_golang/afs/writer.go", "/home/oceanfish81/other_golang/afs/copy_test.go", "/home/oceanfish81/other_golang/afs/faker_test.go", "/home/oceanfish81/other_golang/afs/list_test.go", "/home/oceanfish81/other_golang/afs/move_test.go", "/home/oceanfish81/other_golang/afs/service_test.go", "/home/oceanfish81/other_golang/afs/uploader_test.go", "/home/oceanfish81/other_golang/afs/walker_test.go" ], "NonGoFiles": [], "ImportMap": { "bytes": "bytes", "context": "context", "errors": "errors", "fmt": "fmt", "github.com/pkg/errors": "github.com/pkg/errors", "github.com/stretchr/testify/assert": "github.com/stretchr/testify/assert", "github.com/viant/afs/asset": "github.com/viant/afs/asset", "github.com/viant/afs/base": "github.com/viant/afs/base", "github.com/viant/afs/file": "github.com/viant/afs/file", "github.com/viant/afs/http": "github.com/viant/afs/http", "github.com/viant/afs/matcher": "github.com/viant/afs/matcher", "github.com/viant/afs/mem": "github.com/viant/afs/mem", "github.com/viant/afs/option": "github.com/viant/afs/option", "github.com/viant/afs/scp": "github.com/viant/afs/scp", "github.com/viant/afs/ssh": "github.com/viant/afs/ssh", "github.com/viant/afs/storage": "github.com/viant/afs/storage", "github.com/viant/afs/tar": "github.com/viant/afs/tar", "github.com/viant/afs/url": "github.com/viant/afs/url", "github.com/viant/afs/walker": "github.com/viant/afs/walker", "github.com/viant/afs/zip": "github.com/viant/afs/zip", "io": "io", "io/ioutil": "io/ioutil", "os": "os", "path": "path", "strings": "strings", "sync": "sync", "testing": "testing", "time": "time" }, "PackageFile": { "github.com/pkg/errors": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/pkg/liberrors.a", "github.com/stretchr/testify/assert": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/stretchr/testify/libassert.a", "github.com/viant/afs/asset": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libasset.a", "github.com/viant/afs/base": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libbase.a", "github.com/viant/afs/file": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libfile.a", "github.com/viant/afs/http": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libhttp.a", "github.com/viant/afs/matcher": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libmatcher.a", "github.com/viant/afs/mem": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libmem.a", "github.com/viant/afs/option": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/liboption.a", "github.com/viant/afs/scp": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libscp.a", "github.com/viant/afs/ssh": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libssh.a", "github.com/viant/afs/storage": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libstorage.a", "github.com/viant/afs/tar": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libtar.a", "github.com/viant/afs/url": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/liburl.a", "github.com/viant/afs/walker": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libwalker.a", "github.com/viant/afs/zip": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libzip.a" }, "Standard": { "bytes": true, "context": true, "errors": true, "fmt": true, "io": true, "io/ioutil": true, "os": true, "path": true, "strings": true, "sync": true, "testing": true, "time": true }, "PackageVetx": { "github.com/pkg/errors": "/home/oceanfish81/.cache/go-build/2c/2cb0dea5d565bc000f409b063d0236c03f79d602c0bef46262451652b4331997-d", "github.com/stretchr/testify/assert": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/asset": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/base": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/file": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/http": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/matcher": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/mem": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/option": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/scp": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/ssh": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/storage": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/tar": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/url": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/walker": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/zip": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d" }, "VetxOnly": false, "VetxOutput": "$WORK/b008/vet.out", "SucceedOnTypecheckFailure": false } EOF cd /home/oceanfish81/other_golang/afs TERM='dumb' GCCGO='/usr/local/bin/llvm-goc' /usr/local/tools/vet -atomic -bool -buildtags -errorsas -ifaceassert -nilfunc -printf -stringintconv $WORK/b008/vet.cfg cat >$WORK/b098/vet.cfg << 'EOF' # internal { "ID": "/home/oceanfish81/other_golang/afs_test", "Compiler": "gccgo", "Dir": "/home/oceanfish81/other_golang/afs", "ImportPath": "_/home/oceanfish81/other_golang/afs_test", "GoFiles": [ "/home/oceanfish81/other_golang/afs/example_test.go" > ], "NonGoFiles": [], "ImportMap": { "bytes": "bytes", "context": "context", "fmt": "fmt", "github.com/viant/afs": "github.com/viant/afs", "github.com/viant/afs/asset": "github.com/viant/afs/asset", "github.com/viant/afs/file": "github.com/viant/afs/file", "github.com/viant/afs/matcher": "github.com/viant/afs/matcher", "github.com/viant/afs/option": "github.com/viant/afs/option", "github.com/viant/afs/scp": "github.com/viant/afs/scp", "io": "io", "io/ioutil": "io/ioutil", "log": "log", "path": "path", "strings": "strings" }, "PackageFile": { "github.com/viant/afs": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/libafs.a", "github.com/viant/afs/asset": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libasset.a", "github.com/viant/afs/file": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libfile.a", "github.com/viant/afs/matcher": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libmatcher.a", "github.com/viant/afs/option": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/liboption.a", "github.com/viant/afs/scp": "/home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/viant/afs/libscp.a" }, "Standard": { "bytes": true, "context": true, "fmt": true, "io": true, "io/ioutil": true, "log": true, "path": true, "strings": true }, "PackageVetx": { "github.com/viant/afs": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/asset": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/file": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/matcher": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/option": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d", "github.com/viant/afs/scp": "/home/oceanfish81/.cache/go-build/e3/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-d" }, "VetxOnly": false, "VetxOutput": "$WORK/b098/vet.out", "SucceedOnTypecheckFailure": false } EOF TERM='dumb' GCCGO='/usr/local/bin/llvm-goc' /usr/local/tools/vet -atomic -bool -buildtags -errorsas -ifaceassert -nilfunc -printf -stringintconv $WORK/b098/vet.cfg $WORK/b001/afs.test -test.timeout=10m0s -test.v=true === RUN TestService_Copy copy_test.go:121: failed to lookup key location: [/home/oceanfish81/.secret/id_rsa /home/oceanfish81/.ssh/id_rsa] --- SKIP: TestService_Copy (0.12s) === RUN TestNewFaker --- FAIL: TestNewFaker (0.00s) panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference

    goroutine 18 [running]: testing.tRunner..func2 /home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/testing/testing.go:1112 testing.tRunner..func1 /home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/testing/testing.go:1072 panic /home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/panic.go:712 sync.RWMutex.Lock /home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/sync/rwmutex.go:98 github.x2ecom..z2fviant..z2fafs..z2fmem.Folder.putFile /home/oceanfish81/go/src/github.com/viant/afs/mem/folder.go:60 github.x2ecom..z2fviant..z2fafs..z2fmem.Folder.Put /home/oceanfish81/go/src/github.com/viant/afs/mem/folder.go:178 github.x2ecom..z2fviant..z2fafs..z2fmem.storager.Upload /home/oceanfish81/go/src/github.com/viant/afs/mem/upload.go:61 github.x2ecom..z2fviant..z2fafs..z2fbase.Manager.Upload /home/oceanfish81/go/src/github.com/viant/afs/base/manager.go:102 github.x2ecom..z2fviant..z2fafs..z2fmem.manager.Upload /home/oceanfish81/go/src/github.com/viant/afs/mem/manager.go:44 _..z2fhome..z2foceanfish81..z2fother_golang..z2fafs.service.Upload /home/oceanfish81/other_golang/afs/service.go:66 _..z2fhome..z2foceanfish81..z2fother_golang..z2fafs.TestNewFaker /home/oceanfish81/other_golang/afs/faker_test.go:69 testing.tRunner /home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/testing/testing.go:1163 created by testing.T.Run /home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/testing/testing.go:1214 +0x41c exit status 2 FAIL _/home/oceanfish81/other_golang/afs 1.064s rm -r $WORK/b001/

    CC @thanm , @cherrymui

    Looks like Go files in various sub-folders could be built - but running any tests raises a panic.

    My environment, for gollvm MinSizeRel build

    $ go env && go version GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/oceanfish81/.cache/go-build" GOENV="/home/oceanfish81/.config/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/oceanfish81/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/home/oceanfish81/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/usr/local" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/usr/local/tools" GCCGO="/usr/local/bin/llvm-goc" AR="ar" CC="/usr/bin/clang" CXX="/usr/bin/clang++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build286061473=/tmp/go-build -gno-record-gcc-switches -funwind-tables" go version go1.15.2 gollvm LLVM 12.0.0git linux/amd64

    @adranwit , you can read here

  • No Cache functionality doesn't work with extension URLs.

    No Cache functionality doesn't work with extension URLs.

    When running an operation using the option.NoCache with Source == option.NoCacheBaseURL, the Manager for the extension URL component does not get loaded. This seems unexpected, and prevents updated data sources from updating correctly.

    6c357a82db86aa3c9f75697b08e4d0c691501d45 addresses this.

  • Merge v1

    Merge v1

    • Added WriterProvider interface
    • Changed NewWriter signature (breaking change)
    • Changed Download signature (breaking change, use Open instead)
    • Changed DownloadWithURL signature (breaking change, use OpenURL instead)
    • Renamed Storager.Download to Storage.Open (breaking change)
    • Renamed storage.Downloader to storage.Opener (breaking change)
    • Optimized zip/tar walk streaming
    • Added file.NewWriter
  • Caching is too aggressive

    Caching is too aggressive

    Currently, a lot of data source reads are cached; this may be useful for certain situations, especially if within the API there is support for copying the same data source multiple times.

    However, this cache is too aggressive by default - it will not clear the cache until the process restart. There is not much precedent in infrastructural tools exposing a file-system level API that will inherently cache so aggressively. There may need to be a general system that requires checking if a cache is dirty, or expiring the cache within a reasonable scope.

High Performance, Kubernetes Native Object Storage
High Performance, Kubernetes Native Object Storage

MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag

Jan 2, 2023
a tool for handling file uploads simple

baraka a tool for handling file uploads for http servers makes it easier to make operations with files from the http request. Contents Install Simple

Nov 30, 2022
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site
Bigfile -- a file transfer system that supports http, rpc and ftp protocol   https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

Dec 31, 2022
Go file operations library chasing GNU APIs.
Go file operations library chasing GNU APIs.

flop flop aims to make copying files easier in Go, and is modeled after GNU cp. Most administrators and engineers interact with GNU utilities every da

Nov 10, 2022
Read csv file from go using tags

go-csv-tag Read csv file from Go using tags The project is in maintenance mode. It is kept compatible with changes in the Go ecosystem but no new feat

Nov 16, 2022
File system event notification library on steroids.

notify Filesystem event notification library on steroids. (under active development) Documentation godoc.org/github.com/rjeczalik/notify Installation

Dec 31, 2022
Pluggable, extensible virtual file system for Go

vfs Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as

Jan 3, 2023
An epoll(7)-based file-descriptor multiplexer.

poller Package poller is a file-descriptor multiplexer. Download: go get github.com/npat-efault/poller Package poller is a file-descriptor multiplexer

Sep 25, 2022
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file

QueryCSV enable you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to CSV file

Dec 22, 2021
Goful is a CUI file manager written in Go.
Goful is a CUI file manager written in Go.

Goful Goful is a CUI file manager written in Go. Works on cross-platform such as gnome-terminal and cmd.exe. Displays multiple windows and workspaces.

Dec 28, 2022
Read a tar file contents using go1.16 io/fs abstraction
Read a tar file contents using go1.16 io/fs abstraction

go-tarfs Read a tar file contents using go1.16 io/fs abstraction Usage ⚠️ go-tarfs needs go>=1.16 Install: go get github.com/nlepage/go-tarfs Use: pac

Dec 1, 2022
Open Source Continuous File Synchronization
Open Source Continuous File Synchronization

Goals Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals belo

Jan 9, 2023
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

Jan 1, 2023
The best HTTP Static File Server, write with golang+vue
The best HTTP Static File Server, write with golang+vue

gohttpserver Goal: Make the best HTTP File Server. Features: Human-friendly UI, file uploading support, direct QR-code generation for Apple & Android

Dec 30, 2022
Dragonfly is an intelligent P2P based image and file distribution system.
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

Jan 9, 2023
Fast, dependency-free, small Go package to infer the binary file type based on the magic numbers signature

filetype Small and dependency free Go package to infer file and MIME type checking the magic numbers signature. For SVG file type checking, see go-is-

Jan 3, 2023
📂 Web File Browser
📂 Web File Browser

filebrowser provides a file managing interface within a specified directory and it can be used to upload, delete, preview, rename and edit your files.

Jan 9, 2023
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

Jan 2, 2023