Pluggable, extensible virtual file system for Go

vfs


GitHub tag Build Status GoDoc codecov License: MIT Go Report Card Mentioned in Awesome Go

Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as os, S3, and GCS.

Philosophy

When building our platform, initially we wrote a library that was something to the effect of

      if config.DISK == "S3" {
    	  // do some s3 file system operation
      } else if config.DISK == "mock" {
          // fake something
      } else {
          // do some native os.xxx operation
      }

Not only was ugly but because the behaviors of each "file system" were different and we had to constantly alter the file locations and pass a bucket string (even if the file system didn't know what a bucket was).

We found a handful of third-party libraries that were interesting but none of them had everything we needed/wanted. Of particular inspiration was https://github.com/spf13/afero in its composition of the super-powerful stdlib io.* interfaces. Unfortunately, it didn't support Google Cloud Storage and there was still a lot of passing around of strings and structs. Few, if any, of the vfs-like libraries provided interfaces to easily and confidently create new file system backends.

What we needed/wanted was the following(and more):
  • self-contained set of structs that could be passed around like a file/dir handle
  • the struct would represent an existing or nonexistent file/dir
  • provide common (and only common) functionality across all file system so that after initialization, we don't care what the underlying file system is and can therefore write our code agnostically/portably
  • use io.* interfaces such as io.Reader and io.Writer without needing to call a separate function
  • extensibility to easily add other needed file systems like Microsoft Azure Cloud File Storage
  • prefer native atomic functions when possible (ie S3 to S3 moving would use the native move api call rather than copy-delete)
  • a uniform way of addressing files regardless of file system. This is why we use complete URI's in vfssimple
  • fmt.Stringer interface so that the file struct passed to a log message (or other Stringer use) would show the URI
  • mockable file system
  • pluggability so that third-party implementations of our interfaces could be used

Install

Go install:

go get -u github.com/c2fo/vfs/v5

Usage

We provide vfssimple as basic way of initializing file system backends (see each implementations's docs about authentication). vfssimple pulls in every c2fo/vfs backend. If you need to reduce the backend requirements (and app memory footprint) or add a third party backend, you'll need to implement your own "factory". See backend doc for more info.

You can then use those file systems to initialize locations which you'll be referencing frequently, or initialize files directly

    osFile, err := vfssimple.NewFile("file:///path/to/file.txt")
    s3File, err := vfssimple.NewFile("s3://bucket/prefix/file.txt")

    osLocation, err := vfssimple.NewLocation("file:///tmp/")
    s3Location, err := vfssimple.NewLocation("s3://bucket/")

    osTmpFile, err := osLocation.NewFile("anotherFile.txt") // file at /tmp/anotherFile.txt

You can perform a number of actions without any consideration for the system's api or implementation details.

    osFileExists, err := osFile.Exists() // true, nil
    s3FileExists, err := s3File.Exists() // false, nil
    err = osFile.CopyToFile(s3File) // nil
    s3FileExists, err = s3File.Exists() // true, nil

    movedOsFile, err := osFile.MoveToLocation(osLocation)
    osFileExists, err = osFile.Exists() // false, nil (move actions delete the original file)
    movedOsFileExists, err := movedOsFile.Exists() // true, nil

    s3FileUri := s3File.URI() // s3://bucket/prefix/file.txt
    s3FileName := s3File.Name() // file.txt
    s3FilePath := s3File.Path() // /prefix/file.txt

File's io.* interfaces may be used directly:

    reader := strings.NewReader("Clear is better than clever")
    gsFile, err := vfssimple.NewFile("gs://somebucket/path/to/file.txt")

    byteCount, err := io.Copy(gsFile, reader)
    err := gsFile.Close()

Note: io.Copy() doesn't strictly define what happens if a reader is empty. This is complicated because io.Copy will first delegate actual copying in the following:

  1. if the io.Reader also implements io.WriterTo, WriteTo() will do the copy
  2. if the io.Writer also implements io.ReaderFrom, ReadFrom() will do the copy
  3. finally, if neither 1 or 2, io.Copy will do it's own buffered copy

In case 3, and most implementations of cases 1 and 2, if reader is empty, Write() never gets called. What that means for vfs is there is no way for us to ensure that an empty file does or doesn't get written on an io.Copy(). For instance OS always creates a file, regardless of calling Write() whereas S3 must Write() and Close().

As such, vfs cannot guarantee copy behavior except in our own CopyToFile, MoveToFile, CopyToLocation, and MoveToLocation functions. If you need to ensure a file gets copied/moved with io.Copy(), you must do so yourself OR use vfs's utils.TouchCopy

Third-party Backends

  • none so far

Feel free to send a pull request if you want to add your backend to the list.

See also:

Ideas

Things to add:

  • Add Azure storage backend
  • Provide better List() functionality with more abstracted filtering and paging (iterator?) Return File structs vs URIs?
  • Add better/any context.Context() support

Contributors

Brought to you by the Enterprise Pipeline team at C2FO:

https://github.com/c2fo/

Contributing

1. Fork it (<https://github.com/c2fo/vfs/fork>)
2. Create your feature branch (`git checkout -b feature/fooBar`)
3. Commit your changes (`git commit -am 'Add some fooBar'`)
4. Push to the branch (`git push origin feature/fooBar`)
5. Create a new Pull Request

License

Distributed under the MIT license. See `http://github.com/c2fo/vfs/License.md for more information.

Definitions

absolute path
  • A path is said to be absolute if it provides the entire context need to find a file, including the file system root. An absolute path must begin with a slash and may include . and .. directories.
file path
  • A file path ends with a filename and therefore may not end with a slash. It may be relative or absolute.
location path
  • A location/directory path must end with a slash. It may be relative or absolute.
relative path
  • A relative path is a way to locate a directory or file relative to another directory. A relative path may not begin with a slash but may include . and .. directories.
URI
  • A Uniform Resource Identifier (URI) is a string of characters that unambiguously identifies a particular resource. To guarantee uniformity, all URIs follow a predefined set of syntax rules, but also maintain extensibility through a separately defined hierarchical naming scheme (e.g. http://).

Interfaces

type File

type File interface {
	io.Closer
	io.Reader
	io.Seeker
	io.Writer
	fmt.Stringer

	// Exists returns boolean if the file exists on the file system.  Returns an error, if any.
	Exists() (bool, error)

	// Location returns the vfs.Location for the File.
	Location() Location

	// CopyToLocation will copy the current file to the provided location.
	//
	//   * Upon success, a vfs.File, representing the file at the new location, will be returned.
	//   * In the case of an error, nil is returned for the file.
	//   * CopyToLocation should use native functions when possible within the same scheme.
	//   * If the file already exists at the location, the contents will be overwritten with the current file's contents.
	//   * CopyToLocation will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	CopyToLocation(location Location) (File, error)

	// CopyToFile will copy the current file to the provided file instance.
	//
	//   * In the case of an error, nil is returned for the file.
	//   * CopyToLocation should use native functions when possible within the same scheme.
	//   * If the file already exists, the contents will be overwritten with the current file's contents.
	//   * CopyToFile will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	CopyToFile(file File) error

	// MoveToLocation will move the current file to the provided location.
	//
	//   * If the file already exists at the location, the contents will be overwritten with the current file's contents.
	//   * If the location does not exist, an attempt will be made to create it.
	//   * Upon success, a vfs.File, representing the file at the new location, will be returned.
	//   * In the case of an error, nil is returned for the file.
	//   * When moving within the same Scheme, native move/rename should be used where possible.
	//   * If the file already exists, the contents will be overwritten with the current file's contents.
	//   * MoveToLocation will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	MoveToLocation(location Location) (File, error)

	// MoveToFile will move the current file to the provided file instance.
	//
	//   * If the file already exists, the contents will be overwritten with the current file's contents.
	//   * The current instance of the file will be removed.
	//   * MoveToFile will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	MoveToFile(file File) error

	// Delete unlinks the File on the file system.
	Delete() error

	// LastModified returns the timestamp the file was last modified (as *time.Time).
	LastModified() (*time.Time, error)

	// Size returns the size of the file in bytes.
	Size() (uint64, error)

	// Path returns absolute path, including filename, ie /some/path/to/file.txt
	//
	// If the directory portion of a file is desired, call
	//   someFile.Location().Path()
	Path() string

	// Name returns the base name of the file path.
	//
	// For file:///some/path/to/file.txt, it would return file.txt
	Name() string
	
	// Touch creates a zero-length file on the vfs.File if no File exists.  Update File's last modified timestamp.
    	// Returns error if unable to touch File.
        Touch() error

	// URI returns the fully qualified absolute URI for the File.  IE, s3://bucket/some/path/to/file.txt
	URI() string
}

File represents a file on a file system. A File may or may not actually exist on the file system.

type FileSystem

type FileSystem interface {
	// NewFile initializes a File on the specified volume at path 'absFilePath'.
	//
	//   * Accepts volume and an absolute file path.
	//   * Upon success, a vfs.File, representing the file's new path (location path + file relative path), will be returned.
	//   * On error, nil is returned for the file.
	//   * Note that not all file systems will have a "volume" and will therefore be "":
	//       file:///path/to/file has a volume of "" and name /path/to/file
	//     whereas
	//       s3://mybucket/path/to/file has a volume of "mybucket and name /path/to/file
	//     results in /tmp/dir1/newerdir/file.txt for the final vfs.File path.
	//   * The file may or may not already exist.
	NewFile(volume string, absFilePath string) (File, error)

	// NewLocation initializes a Location on the specified volume with the given path.
	//
	//   * Accepts volume and an absolute location path.
	//   * The file may or may not already exist. Note that on key-store file systems like S3 or GCS, paths never truly exist.
	//   * On error, nil is returned for the location.
	//
	// See NewFile for note on volume.
	NewLocation(volume string, absLocPath string) (Location, error)

	// Name returns the name of the FileSystem ie: Amazon S3, os, Google Cloud Storage, etc.
	Name() string	

	// Scheme returns the uri scheme used by the FileSystem: s3, file, gs, etc.
	Scheme() string

	// Retry will return the retry function to be used by any file system.
	Retry() Retry
}

FileSystem represents a file system with any authentication accounted for.

type Location

type Location interface {
	// String returns the fully qualified absolute URI for the Location.  IE, file://bucket/some/path/
	fmt.Stringer

	// List returns a slice of strings representing the base names of the files found at the Location.
	//
	//   * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	//   * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	//     be checked first.
	List() ([]string, error)

	// ListByPrefix returns a slice of strings representing the base names of the files found in Location whose filenames
	// match the given prefix.
	//
	//   * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	//   * "relative" prefixes are allowed, ie, listByPrefix from "/some/path/" with prefix "to/somepattern" is the same as
	//     location "/some/path/to/" with prefix of "somepattern"
	//   * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	//     be checked first.
	ListByPrefix(prefix string) ([]string, error)

	// ListByRegex returns a slice of strings representing the base names of the files found in Location that matched the
	// given regular expression.
	//
	//   * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	//   * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	//     be checked first.
	ListByRegex(regex *regexp.Regexp) ([]string, error)

	// Volume returns the volume as string. In URI parlance, volume equates to authority.
	// For example s3://mybucket/path/to/file.txt, volume would return "mybucket".
	//
	// Note: Some file systems may not have a volume and will return "".
	Volume() string

	// Path returns absolute location path, ie /some/path/to/.  An absolute path must be resolved to it's shortest path:
	// see path.Clean
	Path() string

	// Exists returns boolean if the location exists on the file system. Returns an error if any.
	Exists() (bool, error)

	// NewLocation is an initializer for a new Location relative to the existing one.
	//
	// Given location:
	//     loc := fs.NewLocation(:s3://mybucket/some/path/to/")
	// calling:
	//     newLoc := loc.NewLocation("../../")
	// would return a new vfs.Location representing:
	//     s3://mybucket/some/
	//
	//   * Accepts a relative location path.
	NewLocation(relLocPath string) (Location, error)

	// Given location:
	// 	   loc := fs.NewLocation("file:///some/path/to/")
	// calling:
	//     loc.ChangeDir("../../")
	// would update the current location instance to
	// file:///some/.
	//
	//   * ChangeDir accepts a relative location path.
	ChangeDir(relLocPath string) error

	//FileSystem returns the underlying vfs.FileSystem struct for Location.
	FileSystem() FileSystem

	// NewFile will instantiate a vfs.File instance at or relative to the current location's path.
	//
	//   * Accepts a relative file path.
	//   * In the case of an error, nil is returned for the file.
	//   * Resultant File path will be the shortest path name equivalent of combining the Location path and relative path, if any.
	//       ie, /tmp/dir1/ as location and relFilePath "newdir/./../newerdir/file.txt"
	//       results in /tmp/dir1/newerdir/file.txt for the final vfs.File path.
	//   * Upon success, a vfs.File, representing the file's new path (location path + file relative path), will be returned.
	//   * The file may or may not already exist.
	NewFile(relFilePath string) (File, error)

	// DeleteFile deletes the file of the given name at the location.
	//
	// This is meant to be a short cut for instantiating a new file and calling delete on that, with all the necessary
	// error handling overhead.
	//
	// * Accepts relative file path.
	DeleteFile(relFilePath string) error

	// URI returns the fully qualified absolute URI for the Location.  IE, s3://bucket/some/path/
	//
	// URI's for locations must always end with a slash.
	URI() string
}

Location represents a file system path which serves as a start point for directory-like functionality. A location may or may not actually exist on the file system.

type Options

type Options interface{}

Options are structs that contain various options specific to the file system

type Retry

type Retry func(wrapped func() error) error

Retry is a function that can be used to wrap any operation into a definable retry operation. The wrapped argument is called by the underlying VFS implementation.

Ex:

    var retrier Retry = func(wrapped func() error) error {
      var ret error
      for i := 0; i < 5; i++ {
         if err := wrapped(); err != nil { ret = err; continue }
      }
      return ret
    }

func DefaultRetryer

func DefaultRetryer() Retry

DefaultRetryer returns a no-op retryer which simply calls the wrapped command without looping.

Comments
  • Using gs.File.CopyToFile after reading the file results in 0 byte file at the destination

    Using gs.File.CopyToFile after reading the file results in 0 byte file at the destination

    This is because the same reader is used for copying and for reading, so by the time you get to CopyToFile the reader is already at the end of the file.

    Probably the simplest solution to this fix will be to just Seek(0,0) before doing the copy.

    As a workaround the seek can be done in user code.

    Any inputs on possible resolutions is welcome, I think it would be good to just call Seek(0, 0) and I will prepare a patch for that.

  • Add support for deleting file versions to s3 and gs

    Add support for deleting file versions to s3 and gs

    Changes Included:

    • file.delete() and location.deleteFile() now support optional argument to specify delete options
    • S3's implementation of file.delete() now removes all the versions of the file object if deleteOptions.DeleteAllVersions is passed in the arguments.
    • GS's implementation of file.delete() now removes all the generations of the file object if deleteOptions.DeleteAllVersions is passed in the arguments.
    • Other implementations of File interface, are noop for deleteOptions.DeleteAllVersions. (Azure implementation is pending)

    Design:

    • In order to keep the delete interface non-passive, decided to change it to a variadic function that takes an interface as argument.
    • If we wish to add additional options for delete, the new options will simply implement the options.DeleteOption interface and add any additional methods that are needed for that option.
    • This approach also allows us to add support for options for other methods as well. For example, if we want to pass file permissions while creating a file, we will just create an interface like FilePermissionsOption and create implementations according to the need.
    • Here is an example that shows the design https://github.com/C2FO/vfs/compare/judd-delete-options
  • File.Write() creates empty files

    File.Write() creates empty files

    Code:

    var Location vfs.Location = initLocation()
    
    func initLocation() vfs.Location {
    
    	if location, err := vfssimple.NewLocation("file:///path/to/public"); err != nil {
    		log.Fatal(err.Error())
    	} else if ok, err := location.Exists(); err != nil {
    		log.Fatal(err.Error())
    	} else if !ok {
    		log.Fatal("Unexpected error")
    	} else {
    		return location
    	}
    
    	return nil
    }
    
    func MakeTestFile() {
    	f, err := Location.NewFile("test.txt")
    
    	if err != nil {
    		log.Fatal(err.Error())
    	}
    
    	if num, err := f.Write([]byte("This is a test text")); err != nil {
    		log.Fatal(err.Error())
    	} else {
    		log.Println("String was writed in file")
    	}
    
    	f.Close()
    }
    

    Problem:

    The function MakeTestFile creates a file without errors, but it is empty.

    System information:

    • Go version: 1.14.1
    • Os/Arch: linux/amd64
    • Package version: 1.6.2
  • How to set S3ForcePathStyle for AWS Client?

    How to set S3ForcePathStyle for AWS Client?

    I'm use Minio as s3 storage. AWS Client can use it only with S3ForcePathStyle=True. s3.Options struct does not have this parameter. Each initialization I have to create a client manually. It is very uncomfortable. It may be possible to add this parameter to the s3.Options?

  • [Feature Request] preserving file metadata

    [Feature Request] preserving file metadata

    Is your feature request related to a problem? Please describe.

    Copy file from system into a local-file-based vfs, I wanna preserve original file MAC timestamp. (last-Modified, last-Accessed, Created)

    Describe the solution you'd like

    Compatible with os.Chtimes(), os.Stat() to get same timestamp as original file.

    Describe alternatives you've considered

    Nothing.

    Additional context

    It's about forensics and e-discovery purpose...

  • Add DeleteAllVersions option support for azure to remove all versions of a file

    Add DeleteAllVersions option support for azure to remove all versions of a file

    Added support for delete all versions in Azure.

    Approach: DeleteAllVersions

    • deletes the blob (this moves the latest version to previous versions)
    • then loops through all the versions of the blob and deletes each version
      • if blobs have soft deletion enabled, this will mark each version as deleted and Azure will permanently delete them as per soft delete policy
      • if blobs don't have soft deletion enabled, this will immediately deletes all the versions
  • EP-5650 -> EP-5650 | Implement Azure Interfaces

    EP-5650 -> EP-5650 | Implement Azure Interfaces

    In order to add a new backend for Azure in VFS we need to implement the FileSystem, Location and File Interfaces in a new Azure package. For access to Azure, see linked SECOPS ticket.

    AC:

    • Docs are created for base implementation
    • Azure package is created
    • Base configs are added
    • FileSystem, Location and File Interfaces has been implemented for Azure
    • FIlesystem config is defined and added, environment config to be added later
  • bug fixes

    bug fixes

    [5.5.4] - 2020-07-06

    Fixed

    • fixes #21 - OS file.Location().ChangeDir(something/) should not affect location
    • fixes #44 - make S3 waitUntilFileExists Exists error more descriptive of root error
    • fixes #45 - fix ASCII diagram alignment for SFTP doc

    Added

    • Improved test coverage for os.FileSystem
  • Fix space issue

    Fix space issue

    Ensure that spaces (url encoded or not) in filenames and paths work as expected when copying (particularly for s3 to s3 native copyObject). Fixes #36

  • EP-2061 -> master | Prep vfs for opensource

    EP-2061 -> master | Prep vfs for opensource

    JIRA Link

    Description

    Our golang vfs library was built with the intention of releasing as opensource. We held off until we could make it prettier and more features but after more than a year in production and only minor changes, we still have yet to release it. Since then google has release a clould agnostic swiss army knife library call go-cloud that implements some vfs-type functionality but nowhere near as full-featured.

    We should prep VFS code for public release. Some steps include:

    • -Ensuring vfs works with the latest third party libs (google-sdk, aws-sdk, etc)- Unsafe to do this until we upgrade to 1.11
    • Ensure docs are up-to-date
    • Consider setting up go.mod (vgo) and/or dep (can these coexist with glide files?)
    • Clean repo? (there are one or two commit messages that contain Jira ticket info).
    • Get sign-off from SecOps and JC
    • Make repo public There may be other items i haven't considered.

    Test Steps

    Put your test steps here....

  • Fixed in-memory implementation edge cases

    Fixed in-memory implementation edge cases

    There were a couple of edge cases that this fixed:

    1. Re-writing data on an in-memory file would previously append data to the existing file rather than overwrite. I fixed this to be consistent with other implementations:
    file.Write([]byte("foo bar")
    file.Close()
    // contents: "foo bar"
    
    file.Write([]byte("foo bar")
    file.Close()
    // contents: "foo barfoo bar"
    
    1. Fixed issue where creating a file in a file system, and then opening it from a relative path in an initialized location yielded no file contents
    fileSystem.NewFile("", "/path/to/some/file.txt")
    file.Write([]byte("foo bar")
    // contents: "foo bar"
    
    location, _ := fileSystem.NewLocation("", "/path/to/")
    f, _ := location.NewFile("some/file.txt")
    ioutil.ReadAll(f) // contents: ""
    
  • Support batch deletion operation in S3 backend

    Support batch deletion operation in S3 backend

    Is your feature request related to a problem? Please describe. Currently, VFS only provides support for deleting individual files. If deleting more than a handful of files, this causes many network roundtrips and is a generally inefficient method of doing things, especially since S3 provides native batch deletion operations.

    We are currently deleting multiple files in a for loop, calling Delete on each one. This is also a waste of money, since we pay for each S3 request.

    Is this something that is on the VFS team's radar?

    Describe the solution you'd like One option is to add a new function DeletePrefix(prefix string) that takes a prefix and deletes all objects matching that prefix in as few API calls as possible.

    Another option would be DeleteMultiple(objects []string) that deletes all objects in string array objects, again using as few API calls as possible. This fits into the existing ListWithPrefix function, and ideally the output of this function could be directly passed into DeleteMultiple

    Describe alternatives you've considered Not including this feature

    Additional context N/A

  • How to specific volume in window in os file?

    How to specific volume in window in os file?

    I using backend package to use local file. Issue is vfs keep the temp file in C drive, because i put my source code in D drive ,so vfs will make a file in D drive. but when i close the file vfs can't rename it back output file due to difference drive.

  • Standardize options handling across providers (pointer to struct vs. pointer)

    Standardize options handling across providers (pointer to struct vs. pointer)

    Right now withOptions under file system takes an empty interface{} that can handle a pointer or a struct.

    There is no code in place to set the options from a pointer to options passed to an interface.

    Also, it seems Azure has a pointer to an Azure option item vs. using the standard interface from vfs. We should look at pros and cons here and standardize.

    These should be addressed in refactor work.

  • Support for opening OS files that are not writable

    Support for opening OS files that are not writable

    Currently OS files are always opened in read/write mode. In cases where files are not writable to the process this causes problems. It would be good to have some way to accomodate read only OS files.

  • Add support for listing all files in a location, including files in

    Add support for listing all files in a location, including files in "subdirectories"

    Currently List() only returns files directly under a specific location. For example, with Google Cloud Storage (gs), if I have f0.txt, f1.txt, d0/f0.txt, d0/f1.txt in location loc, doing loc.List() only returns f0.txt, f1.txt, not files under d0/. It would be nice to have a way to find files with arbitrary depth.

    Possible names for the method could be Location.ListAll(), any other suggestions would be appreciated.

Related tags
Fast extensible file name sanitizer that works in Windows/Linux

Sanity Sanity is a fast and easily extensible file name (and in fact any other string) sanitizer. Usage Built-in rule set Sanity provides a sensible d

Jun 8, 2022
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

Jan 8, 2023
A Small Virtual Filesystem in Go

This is a virtual filesystem I'm coding to teach myself Go in a fun way. I'm documenting it with a collection of Medium posts that you can find here.

Dec 11, 2022
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site
Bigfile -- a file transfer system that supports http, rpc and ftp protocol   https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

Dec 31, 2022
File system event notification library on steroids.

notify Filesystem event notification library on steroids. (under active development) Documentation godoc.org/github.com/rjeczalik/notify Installation

Dec 31, 2022
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

Jan 1, 2023
Dragonfly is an intelligent P2P based image and file distribution system.
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

Jan 9, 2023
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

Jan 2, 2023
File system for GitHub
File system for GitHub

HUBFS · File System for GitHub HUBFS is a read-only file system for GitHub and Git. Git repositories and their contents are represented as regular dir

Dec 28, 2022
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go

GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go Overview GeeseFS allows you to mount an S3 bucket as a file sys

Jan 1, 2023
Encrypted File System in Go

Getting Started: Setup the environment: Install GoLang: $ sudo apt update $ sudo apt upgrade $ sudo apt install libssl-dev gcc pkg-config $ sudo apt

Apr 30, 2022
A rudimentary go program that allows you to mount a mongo database as a FUSE file system

This is a rudimentary go program that allows you to mount a mongo database as a

Dec 29, 2021
Gokrazy mkfs: a program to create an ext4 file system on the gokrazy perm partition

gokrazy mkfs This program is intended to be run on gokrazy only, where it will c

Dec 12, 2022
A FileSystem Abstraction System for Go
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Jan 9, 2023
Experimental typesetting system

ETS Experimental typesetting system This software repository contains a Lua frontend for the typesetting library “Boxes and Glue” which is an algorith

Jan 30, 2022
Go-lang based sonos standup system

Overview This is an CLI tool that can handle timed standup playback on a sonos device. It allows you to add links to audio files that will be randomly

Nov 23, 2021
Abstract File Storage

afs - abstract file storage Please refer to CHANGELOG.md if you encounter breaking changes. Motivation Introduction Usage Matchers Content modifiers S

Dec 30, 2022
a tool for handling file uploads simple

baraka a tool for handling file uploads for http servers makes it easier to make operations with files from the http request. Contents Install Simple

Nov 30, 2022