A user-space file system for interacting with Google Cloud Storage

gcsfuse is a user-space file system for interacting with Google Cloud Storage.

Current status

Please treat gcsfuse as beta-quality software. Use it for whatever you like, but be aware that bugs may lurk, and that we reserve the right to make small backwards-incompatible changes.

The careful user should be sure to read semantics.md for information on how gcsfuse maps file system operations to GCS operations, and especially on surprising behaviors. The list of open issues may also be of interest.

Installing

See installing.md for full installation instructions for Linux and Mac OS X.

Mounting

Prerequisites

  • Before invoking gcsfuse, you must have a GCS bucket that you want to mount. If your bucket doesn't yet exist, create one using the Google Developers Console.
  • GCS credentials are automatically loaded using Google application default credentials, or a JSON key file can be specified explicitly using --key-file. If you haven't already done so, the easiest way to set up your credentials for testing is to run the gcloud tool:
    gcloud auth login

See mounting.md for more information on credentials.

Invoking gcsfuse

To mount a bucket using gcsfuse over an existing directory /path/to/mount, invoke it like this:

gcsfuse my-bucket /path/to/mount

Important: You should run gcsfuse as the user who will be using the file system, not as root. Do not use sudo.

The gcsfuse tool will exit successfully after mounting the file system. Unmount in the usual way for a fuse file system on your operating system:

umount /path/to/mount         # OS X
fusermount -u /path/to/mount  # Linux

If you are mounting a bucket that was populated with objects by some other means besides gcsfuse, you may be interested in the --implicit-dirs flag. See the notes in semantics.md for more information.

See mounting.md for more detail, including notes on running in the foreground and fstab compatibility.

Performance

Latency and rsync

Writing files to and reading files from GCS has a much higher latency than using a local file system. If you are reading or writing one small file at a time, this may cause you to achieve a low throughput to or from GCS. If you want high throughput, you will need to either use larger files to smooth across latency hiccups or read/write multiple files at a time.

Note in particular that this heavily affects rsync, which reads and writes only one file at a time. You might try using gsutil -m rsync to transfer multiple files to or from your bucket in parallel instead of plain rsync with gcsfuse.

Rate limiting

If you would like to rate limit traffic to/from GCS in order to set limits on your GCS spending on behalf of gcsfuse, you can do so:

  • The flag --limit-ops-per-sec controls the rate at which gcsfuse will send requests to GCS.
  • The flag --limit-bytes-per-sec controls the egress bandwidth from gcsfuse to GCS.

All rate limiting is approximate, and is performed over an 8-hour window. By default, requests are limited to 5 per second. There is no limit applied to bandwidth by default.

Upload procedure control

An upload procedure is implemented as a retry loop with exponential backoff for failed requests to the GCS backend. Once the backoff duration exceeds this limit, the retry stops. Flag --max-retry-sleep controls such behavior. The default is 1 minute. A value of 0 disables retries.

GCS round trips

By default, gcsfuse uses two forms of caching to save round trips to GCS, at the cost of consistency guarantees. These caching behaviors can be controlled with the flags --stat-cache-capacity, --stat-cache-ttl and --type-cache-ttl. See semantics.md for more information.

Timeouts

If you are using FUSE for macOS, be aware that by default it will give gcsfuse only 60 seconds to respond to each file system operation. This means that if you write and then flush a large file and your upstream bandwidth is insufficient to write it all to GCS within 60 seconds, your gcsfuse file system may become unresponsive. This behavior can be tuned using the daemon_timeout mount option. See issue #196 for details.

Downloading object contents

Behind the scenes, when a newly-opened file is first modified, gcsfuse downloads the entire backing object's contents from GCS. The contents are stored in a local temporary file whose location is controlled by the flag --temp-dir. Later, when the file is closed or fsync'd, gcsfuse writes the contents of the local file back to GCS as a new object generation.

Files that have not been modified are read portion by portion on demand. gcsfuse uses a heuristic to detect when a file is being read sequentially, and will issue fewer, larger read requests to GCS in this case.

The consequence of this is that gcsfuse is relatively efficient when reading or writing entire large files, but will not be particularly fast for small numbers of random writes within larger files, and to a lesser extent the same is true of small random reads. Performance when copying large files into GCS is comparable to gsutil (see issue #22 for testing notes). There is some overhead due to the staging of data in a local temporary file, as discussed above.

Note that new and modified files are also fully staged in the local temporary directory until they are written out to GCS due to being closed or fsync'd. Therefore the user must ensure that there is enough free space available to handle staged content when writing large files.

Other performance issues

If you notice otherwise unreasonable performance, please file an issue.

Support

gcsfuse is open source software, released under the Apache license. It is distributed as-is, without warranties or conditions of any kind.

For support, visit Server Fault. Tag your questions with gcsfuse and google-cloud-platform, and make sure to look at previous questions and answers before asking a new one. For bugs and feature requests, please file an issue.

Versioning

gcsfuse version numbers are assigned according to Semantic Versioning. Note that the current major version is 0, which means that we reserve the right to make backwards-incompatible changes.

Comments
  • gcsfuse becomes very slow when accessed by multiple threads/processes

    gcsfuse becomes very slow when accessed by multiple threads/processes

    Experimenting with using gcsfuse for DeepVariant (https://github.com/google/deepvariant), I've noticed a big performance degradation when more than one instance is reading the data via gcsfuse (read only).

    Observations:

    • Input data is ~70GB.
    • A single instance processes 1/2048 of the data in 15min when input data is downloaded from gcs.
    • A single instance processes 1/2048 of the data in 12min when input data is fused using gcsfuse.
    • 8 instances (running in parallel in a GCE VM) process 8/2048 of the data in 16min when input data is downloaded from gcs.
    • 8 instances (running in parallel in a GCE VM) process 8/2048 of the data in ~30hours when input data is fused using gcsfuse.

    Access pattern is interleaved, i.e.,

    process 1 reads bytes 1-1000, 20481000 - 20481000 + 1000, so on process 2 reads bytes 2000-3000, 20481000 + 2000 - 2048 1000 + 3000, son .....

    Any idea why the performance is so poor in multiprocess mode? Please let me know if debug info needed. Thanks!

  • gcsfuse hangs when called from systemd startup

    gcsfuse hangs when called from systemd startup

    Automatically mounting via /etc/fstab fails, so I set up a systemd mount unit that looks like this:

    [Unit]
    Description=GCS bucket automount
    Requires=network-online.target
    After=network-online.target
    
    [Mount]
    What=my-gcs-bucket
    Where=/var/gcs
    Type=gcsfuse-wrapper
    Options=rw,user
    
    [Install]
    WantedBy=multi-user.target
    

    gcsfuse-wrapper is a wrapper script that strips away the -n option that /sbin/mount.gcsfuse does not understand. For debugging, it also waits 15 seconds before actually calling /sbin/mount.gcsfuse to ensure that the network is actually up.

    When systemd tries to mount the unit during startup, it hangs.

    root@instance:~# systemctl status var-gcs.mount -l
    ● var-gcs.mount - GCS bucket automount
       Loaded: loaded (/etc/systemd/system/var-gcs.mount; disabled)
       Active: activating (mounting) since Thu 2016-09-08 17:37:05 UTC; 32s ago
        Where: /var/gcs
         What: my-gcs-bucket
      Control: 673 (mount)
       CGroup: /system.slice/var-gcs.mount
               ├─673 /bin/mount -n my-gcs-bucket /var-gcs -t gcsfuse-wrapper -o rw,user
               ├─674 /sbin/mount.gcsfuse-wrapper my-gcs-bucket /var/gcs -n -o rw,noexec,nosuid,nodev,user
               ├─738 sh -c '/sbin/mount.gcsfuse' '-o' 'rw' '-o' 'noexec' '-o' 'nosuid' '-o' 'nodev' '-o' 'user' 'my-gcs-bucket' '/var/gcs'
               ├─739 /sbin/mount.gcsfuse -o rw -o noexec -o nosuid -o nodev -o user my-gcs-bucket /var/gcs
               ├─742 gcsfuse -o noexec -o nosuid -o nodev -o rw my-gcs-bucket /var/gcs
               └─749 /usr/bin/gcsfuse --foreground -o noexec -o nosuid -o nodev -o rw my-gcs-bucket /var/gcs
    
    Sep 08 17:37:05 instance systemd[1]: Mounting GCS bucket automount...
    Sep 08 17:37:05 instance mount[673]: [wrapper] Sleeping for 15 seconds to wait for the network to come up...
    Sep 08 17:37:21 instance mount[673]: [wrapper] Invoking gcsfuse with: '/sbin/mount.gcsfuse' '-o' 'rw' '-o' 'noexec' '-o' 'nosuid' '-o' 'nodev' '-o' 'user' 'my-gcs-bucket' '/var/gcs'
    Sep 08 17:37:21 instance mount[673]: Calling gcsfuse with arguments: -o noexec -o nosuid -o nodev -o rw my-gcs-bucket /var/gcs
    Sep 08 17:37:21 instance mount[673]: Using mount point: /var/gcs
    Sep 08 17:37:21 instance mount[673]: Opening GCS connection...
    Sep 08 17:37:21 instance mount[673]: Opening bucket...
    Sep 08 17:37:21 instance mount[673]: Mounting file system...
    

    After a while, systemd will timeout and terminate the process. If I issue a systemctl restart var-gcs.mount, it will work just fine and properly mount the bucket.

    Unfortunately, I was unable to make /sbin/mount.gcsfuse accept the --debug_* options, so I cannot provide more detailed information.

    What could be causing this, and how can I work around it? Or is there a better way to automatically mount the bucket on boot?

  • unbearable slowness of directory listings

    unbearable slowness of directory listings

    500 files in a directory. the instance is in us-east and the storage is there as well (regional).

    > time gsutil ls gs:\\mybucketdir | wc -l
    
    500
    real	0m0.974s
    user	0m0.484s
    sys	0m0.156s
    
    

    When running ls /mnt/mymountpoint

    I get the following timings

    gcs: Req             0x98: -> ListObjects() (199.70033ms): OK
    gcs: Req             0x9a: <- ListObjects()
    gcs: Req             0x9a: -> ListObjects() (203.140236ms): OK
    gcs: Req             0x9b: <- ListObjects()
    gcs: Req             0x9b: -> ListObjects() (197.373822ms): OK
    gcs: Req             0x9c: <- ListObjects()
    gcs: Req             0x9c: -> ListObjects() (184.619554ms): OK
    gcs: Req             0x9d: <- ListObjects()
    gcs: Req             0x9d: -> ListObjects() (764.47926ms): OK
    gcs: Req             0x9e: <- ListObjects()
    gcs: Req             0x9e: -> ListObjects() (150.530204ms): OK
    gcs: Req             0x9f: <- ListObjects()
    gcs: Req             0x9f: -> ListObjects() (199.542155ms): OK
    gcs: Req             0xa0: <- ListObjects()
    gcs: Req             0xa0: -> ListObjects() (181.239533ms): OK
    gcs: Req             0xa1: <- ListObjects()
    gcs: Req             0xa1: -> ListObjects() (215.307041ms): OK
    gcs: Req             0xa2: <- ListObjects()
    gcs: Req             0xa2: -> ListObjects() (205.366617ms): OK
    gcs: Req             0xa3: <- ListObjects()
    gcs: Req             0xa3: -> ListObjects() (197.569194ms): OK
    gcs: Req             0xa4: <- ListObjects()
    gcs: Req             0xa4: -> ListObjects() (197.323209ms): OK
    gcs: Req             0xa5: <- ListObjects()
    gcs: Req             0xa5: -> ListObjects() (200.654838ms): OK
    gcs: Req             0xa6: <- ListObjects()
    gcs: Req             0xa6: -> ListObjects() (163.866557ms): OK
    gcs: Req             0xa7: <- ListObjects()
    gcs: Req             0xa7: -> ListObjects() (217.490382ms): OK
    gcs: Req             0xa8: <- ListObjects()
    gcs: Req             0xa8: -> ListObjects() (225.55158ms): OK
    gcs: Req             0xa9: <- ListObjects()
    gcs: Req             0xa9: -> ListObjects() (188.412506ms): OK
    gcs: Req             0xaa: <- ListObjects()
    gcs: Req             0xaa: -> ListObjects() (201.611923ms): OK
    gcs: Req             0xab: <- ListObjects()
    gcs: Req             0xab: -> ListObjects() (203.075812ms): OK
    

    it's mounted with these options: --foreground --limit-ops-per-sec -1 --implicit-dirs -o ro -o allow_other --uid 1001 --gid 1002 --debug_gcs Is there any way to make it faster? it takes few minutes to run ls compared to gsutil that takes few seconds.

  • uploading files via gcsfuse result in 0 file size

    uploading files via gcsfuse result in 0 file size

    Hello Aaron,

    We run into a bug in gcsfuse: when we try to upload data via gcsfuse (wget) files have 0 size.

    We are using this version of gcsfuse:

    gcsfuse --version
    gcsfuse version 0.20.0 (Go version go1.7.1)
    

    Bucket is mounted with this command: /usr/bin/gcsfuse --foreground --debug_gcs --debug_fuse --key-file=/srv/buckets_keys/user747_key.json --implicit-dirs --dir-mode=775 --file-mode=775 -o allow_other --gid=1005 --uid=100747 --temp-dir=/var/tmp idna_prod_data_user747 /data/user747

    Trying to get a file: wget http://speedtest.london.linode.com/100MB-london.bin.4

    Debug output info:

    fuse_debug: Op 0x0000636b        connection.go:474] -> OK
    fuse_debug: Op 0x0000636c        connection.go:395] <- WriteFile (inode 6, handle 0, offset 104162970, 4096 bytes)
    fuse_debug: Op 0x0000636c        connection.go:474] -> OK
    fuse_debug: Op 0x0000636d        connection.go:395] <- WriteFile (inode 6, handle 0, offset 104167066, 4096 bytes)
    fuse_debug: Op 0x0000636d        connection.go:474] -> OK
    fuse_debug: Op 0x0000636e        connection.go:395] <- WriteFile (inode 6, handle 0, offset 104171162, 4096 bytes)
    fuse_debug: Op 0x0000636e        connection.go:474] -> OK
    fuse_debug: Op 0x0000636f        connection.go:395] <- WriteFile (inode 6, handle 0, offset 104175258, 4096 bytes)
    fuse_debug: Op 0x0000636f        connection.go:474] -> OK
    fuse_debug: Op 0x00006370        connection.go:395] <- WriteFile (inode 6, handle 0, offset 104179354, 4096 bytes)
    fuse_debug: Op 0x00006370        connection.go:474] -> OK
    fuse_debug: Op 0x00006371        connection.go:395] <- WriteFile (inode 6, handle 0, offsgcs: Req             0x16: -> CreateObject("100MB-london.bin.4") (1.864889607s): gcs.PreconditionError: googleapi: Error 412: Precondition Failed, conditionNotMet
    fuse_debug: Op 0x00006416        connection.go:474] -> OK
    fuse_debug: Op 0x00006417        connection.go:395] <- ReleaseFileHandle
    fuse_debug: Op 0x00006418        connection.go:395] <- GetInodeAttributes (inode 1)
    fuse_debug: Op 0x00006418        connection.go:474] -> OK
    fuse_debug: Op 0x00006417        connection.go:474] -> OK
    fuse_debug: Op 0x00006419        connection.go:395] <- LookUpInode (parent 1, name "100MB-london.bin.4")
    gcs: Req             0x17: <- StatObject("100MB-london.bin.4")
    gcs: Req             0x17: -> StatObject("100MB-london.bin.4") (27.90176ms): OK
    fuse_debug: Op 0x00006419        connection.go:474] -> OK
    fuse_debug: Op 0x0000641a        connection.go:395] <- ForgetInode (inode 7)
    fuse_debug: Op 0x0000641a        connection.go:474] -> OK
    fuse_debug: Op 0x0000641b        connection.go:395] <- LookUpInode (parent 1, name "100MB-london.bin.4")
    fuse_debug: Op 0x0000641c        connection.go:395] <- ForgetInode (inode 6)
    fuse_debug: Op 0x0000641b        connection.go:474] -> OK
    fuse_debug: Op 0x0000641d        connection.go:395] <- SetInodeAttributes (inode 8, atime 2016-11-10 10:54:31 +0000 UTC, mtime 2016-03-29 21:45:45 +0000 UTC)
    gcs: Req             0x18: <- UpdateObject("100MB-london.bin.4")
    fuse_debug: Op 0x0000641c        connection.go:474] -> OK
    gcs: Req             0x18: -> UpdateObject("100MB-london.bin.4") (162.179136ms): OK
    fuse_debug: Op 0x0000641d        connection.go:474] -> OK
    fuse_debug: Op 0x0000641e        connection.go:395] <- unknown (inode 8, opcode 24)
    fuse_debug: Op 0x0000641e        connection.go:476] -> Error: "function not implemented"
    
    

    Checking file size:

    # ls -lah 100MB-london.bin.4
    -rwxrwxr-x 1 user747 web 0 Mar 29  2016 100MB-london.bin.4
    
    

    https://developers.google.com/doubleclick-search/v2/standard-error-responses#PRECONDITION_FAILED

    Why is this happening and what can we do to avoid it?

    Thanks a lot in advance, Anna

  • panic: runtime error: invalid memory address or nil pointer dereference

    panic: runtime error: invalid memory address or nil pointer dereference

    We are runnung gcsfuse 0.17.1 on Linux. After using the filesystem for some time, it gets inaccessible, and the log says (filenames hidden for security-reasons):

    gcs: Req           0x4086: <- Read("FILENAME_HIDDEN", <nil>)
    gcs: Req           0x4086: -> Read("FILENAME_HIDDEN", <nil>) (46.366899ms): OK
    gcs: Req           0x4087: <- StatObject("FILENAME_HIDDEN_2/")
    gcs: Req           0x4087: -> StatObject("FILENAME_HIDDEN_2/") (36.40526ms): gcs.NotFoundError: googleapi: Error 404: Not Found, notFound
    gcs: Req           0x4088: <- StatObject("FILENAME_HIDDEN_2")
    gcs: Req           0x4088: -> StatObject("FILENAME_HIDDEN_2") (38.451819ms): OK
    gcs: Req           0x4089: <- StatObject("FILENAME_HIDDEN_3/")
    gcs: Req           0x4089: -> StatObject("FILENAME_HIDDEN_3/") (37.150623ms): OK
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal 0xb code=0x1 addr=0x48 pc=0x52a44f]
    
    goroutine 644219 [running]:
    panic(0x8f14c0, 0xc8200100d0)
            /home/jacobsa/clients/go/src/runtime/panic.go:464 +0x3e6
    github.com/googlecloudplatform/gcsfuse/internal/fs.(*fileSystem).ForgetInode(0xc820139e00, 0x7ff0c6fa3858, 0xc820d66a80, 0xc8203d2280, 0x0, 0x0)
            /tmp/build_gcsfuse_gopath707641970/src/github.com/googlecloudplatform/gcsfuse/internal/fs/fs.go:1015 +0xbf
    github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc8203d7a00, 0xc820139ec0, 0x7ff0c6fa3858, 0xc820d66a80, 0x7f4920, 0xc8203d2280)
            /tmp/build_gcsfuse_gopath707641970/src/github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:137 +0x77e
    created by github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps
            /tmp/build_gcsfuse_gopath707641970/src/github.com/googlecloudplatform/gcsfuse/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:108 +0x18f
    

    What can we do?

  • Mount a single dir from bucket.

    Mount a single dir from bucket.

    Hi Aaron. Thank for you work. Gcsfuse can't mount a single dir from bucket, only whole bucket can be mounted. Can you add this feature, it would be useful in some situations.

  • Mounting a sub folder of a bucket.

    Mounting a sub folder of a bucket.

    Hi team, I was trying to mount a directory from a compute engine instance onto a google bucket, which I am successfull. But what i am trying to do is, having files directly go into the bucket via sftp from a remote machine, so the remote machine user gets to see the bucket as a folder and the objects inside the bucket as subfolders. As i will be getting files from multiple remote machines onto the same bucket but to different folders under the bucket, wanted to know if there is a way to mount a object inside the bucket to a user so that the user gets to see only that folder and not list everything that's there in the bucket.

    eg: gcsfuse --temp dir /data mybucket/subfolder /foldertomount/

    currently we can mount the bucket as a whole, and not a particular folder inside the bucket.

  • System file type/ProxMox Backup Server

    System file type/ProxMox Backup Server

    Hello,

    I have mounted GCS with gcsfuse to ProxMox Backup Server /Debian, however I am not able to read/write into it, I am getting the bellow error:

    mkdir: cannot create directory ‘/mnt/GCPStorage/Test’: Input/output error

    I thin ProxMox is not able to read /write to the mount since the fs type is "fuse.gcsfuse " Could you please help me,

  • Error mounting bucket with 0.31

    Error mounting bucket with 0.31

    Since 0.31 came out our projects using gcsfuse built from sources can't mount buckets :

    Exec lifecycle hook ([gcsfuse --key-file /root/secret/key.json -o allow_other,nonempty,rw --uid 33 --gid 33 xxxx /var/www/html/web/uploads]) for Container "xxxx-api-develop" in Pod "xxxx-api-develop-5694d6d65c-q5p8q_dev(fbe997c4-ccd0-11ea-b864-42010a840013)" failed - error: command 'gcsfuse --key-file /root/secret/key.json -o allow_other,nonempty,rw --uid 33 --gid 33 xxxx /var/www/html/web/uploads' exited with 1: daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: Mount: mount: invalid argument , message: "Using mount point: /var/www/html/web/uploads\nOpening GCS connection...\nOpening bucket...\nMounting file system...\ndaemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: Mount: mount: invalid argument\n

    The mount arguments have not changed, it had been working for months

  • Input/Output errors

    Input/Output errors

    I just installed gcsfuse on a "small" VM instance, created a directory and a bucket, then mounted the bucket. I can see the file in the bucket, but when I try to create a new file from the VM, I get Input/Output error:

    $ gcsfuse -v
    gcsfuse version 0.18.2 (Go version go1.6.2)
    $ mkdir swingleydev_vm
    $ gcsfuse swingleydev_vm /home/cswingle/swingleydev_vm
    Using mount point: /home/cswingle/swingleydev_vm
    Opening GCS connection...
    Opening bucket...
    Mounting file system...
    File system has been successfully mounted.
    $ ls swingleydev_vm/
    README.rst
    $ touch swingleydev_vm/foo.txt
    touch: cannot touch ‘swingleydev_vm/foo.txt’: Input/output error
    

    Here's the mount info:

    $ cat /proc/mounts | grep swingleydev
    swingleydev_vm /home/cswingle/swingleydev_vm fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1001,default_permissions 0 0
    
  • Run integration tests from GCE

    Run integration tests from GCE

    Want to try out the integration tests for all of the following packages, on a GCE instance in the US:

    • jacobsa/gcloud
    • jacobsa/fuse
    • googlecloudplatform/gcsfuse

    Things might work differently with differing network latency and listing latency conditions.

  • Web Server(Nginx) can not write in mount dir

    Web Server(Nginx) can not write in mount dir

    Hello. I mounted the bucket to compute engine vm via gcsfuse and then ran the nginx program in the vm. Normally the nginx program writes the cache to the directory, i.e. the mounted bucket, using the proxy_cache directive. However, I have a problem that nginx can create cache files in the filesystem under the bucket directory, but the size of the cache file is always 0B. client requests keep getting "Miss cache" status. So it seems that after mounting with gcsfuse, the Nginx application can only create cache files, but cannot write to them.

    My vm environment is. Machine type: n2-standard-2 CPU platform: Intel Cascade Lake Architecture: x86/64 System: ubuntu-1804

    In addition, gcsfuse has specified a service account with owner privileges via the --key-file directive, and the nginx program has been run with the root user.

    For example, the following debug log is an empty file (8/2b/0a7ed71cddee71b9276a0d72a030f2b8) created in the bucket after a client request and not written to the cache. What could be the cause of this possibility?

  • [Don't Review] Build Script to Run Pytorch Model on DLVM Instance

    [Don't Review] Build Script to Run Pytorch Model on DLVM Instance

    Have to resolve issues: (a) Pytorch cache issues - when we run the first time (b) Error handling to make this script idempotent. (c) Resolve root/non-root issue, since data on the gcsfuse is currenly accessed by root user. (d) Pending discussion around - log file creation and (Kernel cache bypassing code)

  • Allow opting-in for no credentials mount

    Allow opting-in for no credentials mount

    Already mentioned in this issue, but got eventually buried.

    The problem is that mounting public buckets via gcsfuse requires some valid credentials present even though REST API doesn't. It would be great to have some opt-in flag like --allow-anonymous, which will prevent failing in case of nil tokenSrc, but rather create an anonymous connection (maybe with some warning).

  • Mounting bucket via fstab hangs

    Mounting bucket via fstab hangs

    Describe the bug Mounting bucket via fstab hangs

    To Collect more Debug logs Steps to reproduce the behavior:

    $ cat /etc/fstab
    ...
    service-dataset-bucket-v2  /mnt/dataset      gcsfuse ro,allow_other,noauto
    
    $ mount /mnt/dataset
    Calling gcsfuse with arguments: -o ro -o allow_other service-dataset-bucket-v2 /mnt/dataset
    2022/12/01 14:10:16.381445 Start gcsfuse/0.41.9 (Go version go1.18.4) for app "" using mount point: /mnt/dataset
    2022/12/01 14:10:16.388247 Opening GCS connection...
    # hangs forever
    

    Several gcsfuse processes hanging:

    pgrep -a gcsfuse
    
    45643 /sbin/mount.gcsfuse service-dataset-bucket-v2 /mnt/dataset -o ro,allow_other
    45648 gcsfuse -o ro -o allow_other service-dataset-bucket-v2 /mnt/dataset
    45660 /usr/bin/gcsfuse --foreground -o ro -o allow_other service-dataset-bucket-v2 /mnt/dataset
    

    System (please complete the following information):

    • OS: Ubuntu 22.04
    • Platform: native
    • Version: 0.41.9 (Go version go1.18.4)

    Additional context If I run arguments from logs above ("Calling gcsfuse with arguments") from user space, it mounts ok:

    gcsfuse -o ro -o allow_other service-dataset-bucket-v2 /mnt/datas
    
Related tags
SeaweedFS a fast distributed storage system for blobs, objects, files, and data lake, for billions of files
SeaweedFS a fast distributed storage system for blobs, objects, files, and data lake, for billions of files

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

Jan 8, 2023
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

Jan 2, 2023
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

Jan 1, 2023
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go

Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go Overview Goofys allows you to mount an S3 bucket as a filey system. It's a

Jan 8, 2023
The Swift Virtual File System

*** This project is not maintained anymore *** The Swift Virtual File System SVFS is a Virtual File System over Openstack Swift built upon fuse. It is

Dec 11, 2022
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

Aug 7, 2017
A FileSystem Abstraction System for Go
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Dec 31, 2022
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr

Jan 9, 2023
This is a simple file storage server. User can upload file, delete file and list file on the server.
This is a simple file storage server.  User can upload file,  delete file and list file on the server.

Simple File Storage Server This is a simple file storage server. User can upload file, delete file and list file on the server. If you want to build a

Jan 19, 2022
provide api for cloud service like aliyun, aws, google cloud, tencent cloud, huawei cloud and so on

cloud-fitter 云适配 Communicate with public and private clouds conveniently by a set of apis. 用一套接口,便捷地访问各类公有云和私有云 对接计划 内部筹备中,后续开放,有需求欢迎联系。 开发者社区 开发者社区文档

Dec 20, 2022
An encrypted object storage system with unlimited space backed by Telegram.

TGStore An encrypted object storage system with unlimited space backed by Telegram. Please only upload what you really need to upload, don't abuse any

Nov 28, 2022
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

Jan 8, 2023
Contentrouter - Protect static content via Firebase Hosting with Cloud Run and Google Cloud Storage

contentrouter A Cloud Run service to gate static content stored in Google Cloud

Jan 2, 2022
TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage

TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage. By leveraging smart contracts, client-side encryption, and sophisticated redundancy (via Reed-Solomon codes), TurtleDex allows users to safely store their data with hosts that they do not know or trust.

May 29, 2021
TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage.

TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage. By leveraging smart contracts, client-side e

Feb 17, 2021
Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.
Rclone (

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Jan 5, 2023
Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.
Rclone (

Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr

Nov 5, 2021
GoDrive: A cloud storage system similar to Dropbox or Google Drive, with resilient
GoDrive: A cloud storage system similar to Dropbox or Google Drive, with resilient

Cloud Storage Service Author: Marisa Tania, Ryan Tjakrakartadinata Professor: Matthew Malensek See project spec here: https://www.cs.usfca.edu/~mmalen

Dec 7, 2021