File system event notification library on steroids.

Comments
  • Life is not easy - rework Event type

    Life is not easy - rework Event type

    This commit changes Event type and makes it quite similar to Signal types/interface from std library:

    os - defines a Signal interface.
    os/signal - uses os.Signal interface.
    syscall - defines and declares Signal type which fulfills os.Signal interface. The syscall.Signal type has different impl for each platform.
    
    os - should define an Event (Notify) interface
    os/notify - should contain the rjmagicwtf implementation which uses os.Event interface + platform dependent impl for watchers.
    syscall - defines Event type per each platform...
    

    I didn't create an Event interface because I'm too lazy to fix @rjeczalik 's code which will break after such changes xD

  • epoll_wait syscall doesn't exist on arm64.

    epoll_wait syscall doesn't exist on arm64.

    Implement it with by calling epoll_pwait(). According to man epoll_pwait, calling epoll_pwait with sigmask of NULL is identical to epoll_wait.

    Bring the fixes from golang.org/x/sys/unix to support these transparently on arm64.

    This is needed to fix https://github.com/minio/mc/issues/2047

  • Error handling

    Error handling

    1. Close channel and then notify.Error(ch) error:
    ch := make(chan notify.EventInfo)
    notify.Watch("/home/pknap/.secretporndir", ch, notify.Delete)
    switch ei, ok := <-ch; {
      case ei.Event() == notify.Delete:
        fmt.Printf("Who deleted my %s file?;/\n", ei.Name())
      case !ok:
       fmt.Println("wtf: %v", notify.Error(ch))
    }
    

    [edit] The 1 option can look better:

    ch := make(chan notify.EventInfo)
    notify.Watch("/home/pknap/.secretporndir", ch, notify.Delete)
    switch ei, ok := <-ch; ei.Event() {
      case notify.Delete:
        fmt.Printf("Who deleted my %s file?;/\n", ei.Name())
      default:
        if !ok {
           fmt.Println("wtf: %v", notify.Error(ch))
        }
    
    }
    
    1. Magic unexported event info + close chan(?):
    type notifyError error
    func (e *notifyError) Event() Event     { ... }
    func (e *notifyError) IsDir() bool       { ... }
    func (e *notifyError) Name() string     { ... }
    func (e *notifyError) Sys() interface{}  { ... } 
    func (e *notifyError) Err() error { ... } // Extra method:>
    

    then:

    ch := make(chan notify.EventInfo)
    notify.Watch("/home/pknap/rottenbodies", ch, notify.Delete)
    switch ei := <-ch; ei.Event() {
      case notify.Delete:
        fmt.Printf("Who deleted my %s file?;/\n", ei.Name())
      default: // need to have all registered events in case statement or check `ei.Err() != nil` after `default`
       fmt.Println("wtf: %v", ei.Err())
    }
    

    Other ideas?

  • incompatible with Go 1.6 cgo/GC rules

    incompatible with Go 1.6 cgo/GC rules

    See https://tip.golang.org/doc/go1.6#cgo for background.

    Using Go 1.6rc2 with github.com/cortesi/modd immediately generates a panic from within this library. To reproduce:

    • Use Go 1.6rc2.
    • go get -u github.com/cortesi/modd/cmd/modd
    • Write a file called modd.conf that contains * {}.
    • Execute modd.

    Result:

    panic: runtime error: cgo argument has Go pointer to Go pointer
    
    goroutine 1 [running]:
    github.com/rjeczalik/notify.(*stream).Start(0xc820012700, 0x0, 0x0)
        $GOPATH/src/github.com/rjeczalik/notify/watcher_fsevents_cgo.go:131 +0x270
    github.com/rjeczalik/notify.(*fsevents).watch(0xc8200742c0, 0xc82000e7e0, 0x17, 0x100001b00, 0x0, 0x0)
        $GOPATH/src/github.com/rjeczalik/notify/watcher_fsevents.go:206 +0x313
    github.com/rjeczalik/notify.(*fsevents).RecursiveWatch(0xc8200742c0, 0xc82000e7e0, 0x17, 0x1b0000001b00, 0x0, 0x0)
        $GOPATH/src/github.com/rjeczalik/notify/watcher_fsevents.go:258 +0x51
    go.(*struct { github.com/rjeczalik/notify.watcher; github.com/rjeczalik/notify.recursiveWatcher }).RecursiveWatch(0xc820086140, 0xc82000e7e0, 0x17, 0xc800001b00, 0x0, 0x0)
        <autogenerated>:50 +0x7c
    github.com/rjeczalik/notify.(*recursiveTree).Watch(0xc8200880a0, 0xc82000e7e0, 0x17, 0xc8200a86c0, 0xc82000aaa8, 0x1, 0x1, 0x0, 0x0)
        $GOPATH/src/github.com/rjeczalik/notify/tree_recursive.go:286 +0xad6
    github.com/rjeczalik/notify.Watch(0xc82000aaa0, 0x5, 0xc8200a86c0, 0xc82000aaa8, 0x1, 0x1, 0x0, 0x0)
        $GOPATH/src/github.com/rjeczalik/notify/notify.go:64 +0x7d
    github.com/cortesi/modd/watch.Watch(0xc82000e780, 0x2, 0x2, 0x5f5e100, 0xc8200a84e0, 0xc82000e780, 0x0, 0x0)
        $GOPATH/src/github.com/cortesi/modd/watch/watch.go:301 +0x145
    github.com/cortesi/modd.runOnChan(0xc8200a84e0, 0x4383390, 0x4b002f0, 0xc820012600, 0xc82000e680, 0x4307bb0, 0xb, 0xc82004fdb8, 0x0, 0x0, ...)
        $GOPATH/src/github.com/cortesi/modd/modd.go:89 +0x4d6
    github.com/cortesi/modd.Run(0x4b002f0, 0xc820012600, 0xc82000e680, 0x4307bb0, 0xb, 0xc82004fdb8, 0x0, 0x0, 0x4013277, 0x0, ...)
        $GOPATH/src/github.com/cortesi/modd/modd.go:141 +0xbe
    main.main()
        $GOPATH/src/github.com/cortesi/modd/cmd/modd/main.go:102 +0x9a2
    
  • Missing events when creating files by bulk

    Missing events when creating files by bulk

    Hey Rafal, I'm running into a weird issue with watching, where lot of events are dropped when the files are created in a rapid succession, ie using for f in {0..10}; do echo $f > "$f.txt"; done

    Here's the code I'm using:

    package main
    
    import (
            "github.com/rjeczalik/notify"
            "log"
    )
    
    func main() {
           c := make(chan notify.EventInfo)
            if err := notify.Watch("<folder>", c, notify.All); err != nil {
                log.Fatal(err)
            }
            defer notify.Stop(c)
    
            var index = 0
            for {
                    ei := <-c
                    index += 1
                    if ei.Event() == notify.Create{
                            log.Println("Got event:", ei, index)
                    }
            }
    }
    

    I tried the same with fsnotify and I'm not noticing any drop there:

    package main
    
    import (
            "gopkg.in/fsnotify.v1"
            "log"
    )
    
    func main() {
            watcher, err := fsnotify.NewWatcher()
            if err != nil {
                log.Fatal(err)
            }
            defer watcher.Close()
    
            var index = 0
            done := make(chan bool)
            go func() {
                for {
                    select {
                    case event := <-watcher.Events:
                        if event.Op&fsnotify.Write == fsnotify.Write {
                            index += 1
                            log.Println("modified file:", event.Name, index)
                        }
                    case err := <-watcher.Errors:
                        log.Println("error:", err)
                    }
                }
            }()
    
            if err = watcher.Add("<folder>"); err != nil {
                log.Fatal(err)
            }
            <-done
    }
    

    If I added a bit of sleep (0.1s) between file creation, all the events are received. I tested it on Ubuntu (Koding VM).

    Regards.

  • Refactor inotify globals to be a part of struct

    Refactor inotify globals to be a part of struct

    The issue is incomming integration tests would need to instantiate and terminate multiple watcher instances via NewWatcher. As long as inotify is implemented using single global and single syscall.InotifyInit this would break those tests.

    There is no guarantee that integration tests for error handling would be capable to do full and proper cleanup of the global inotify instance, and it's obvious what happens next when a test is not idempotent to a Watcher.

    Moreover running tests in parallel would not be possible without it.

  • FSEvents: lock the CF run loop goroutine to its thread

    FSEvents: lock the CF run loop goroutine to its thread

    CoreFoundation run loops are tied to a particular thread. The goroutine that sets up the run loop may get rescheduled on a different thread just after adding the source. In that case the run loop that's started wouldn't have any sources and thus exit immediately.

    Fix this by locking the goroutine to its thread. The thread is wasted on the run loop anyway, so we might as well claim it earlier.

    Should help with #139

  • Some fixes regarding unwatching and rewatching a path

    Some fixes regarding unwatching and rewatching a path

    This started in issue #124 and replaces the previously split up PRs. This is bundled as the fixed problems come up in the same tests, but are in separate commits to make each change-diff about a single problem. The commits should explain what is going on.

  • FreeBSD - ZFS mounted drive

    FreeBSD - ZFS mounted drive

    Hi, I am trying to use notify to listen to CREATE events in a volume mounted over ZFS in a server running FreeBSD.

    But the event does not get captured at all. The code works fine in my Ubuntu distribution.

    Any idea why? Thanks!

  • Watcher: extend Unwatch arguments with `Event` to be unwatched.

    Watcher: extend Unwatch arguments with `Event` to be unwatched.

    This way you can stop watching only specified events, what is necessary and makes it work as comment says.

    It does not change how current implementations work (they ignore the value, and passed is always All), but allows it to be utilized.

  • Revert

    Revert "Handle pointers that are now integers in Go 1.10 (fixes #135)"

    The thing I did to fix compatibility for Go 1.10 has been changed again on the Go tree since. To actually build with go1.10rc1 we need to revert my thing. Sorry.

    This reverts commit ff2d4d2cedc09db23cc46d3eeb8b402bca6d819d.

  • C.FSEventStreamScheduleWithRunLoop(ref, runloop, C.kCFRunLoopDefaultMode) MacOS 13 future remove

    C.FSEventStreamScheduleWithRunLoop(ref, runloop, C.kCFRunLoopDefaultMode) MacOS 13 future remove

    cgo-gcc-prolog:217:2: warning: 'FSEventStreamScheduleWithRunLoop' is deprecated: first deprecated in macOS 13.0 - Use FSEventStreamSetDispatchQueue instead. [-Wdeprecated-declarations]

    FSEventStreamScheduleWithRunLoop(
      FSEventStreamRef   streamRef,
      CFRunLoopRef       runLoop,
      CFStringRef        runLoopMode) API_DEPRECATED("Use FSEventStreamSetDispatchQueue instead.", macos(10.5, 13.0), ios(6.0,16.0));
    
    extern void 
    FSEventStreamSetDispatchQueue(
      FSEventStreamRef              streamRef,
      dispatch_queue_t __nullable   q)                                      __OSX_AVAILABLE_STARTING(__MAC_10_6, __IPHONE_6_0);
    

    Sadly, I tried to modify it and it didn't work

  • Fixes for recursive tree watcher

    Fixes for recursive tree watcher

    Hello! This PR fixes 3 problems we encountered in the recursive tree watcher implementation and adds 2 new tests:

    • In the recursive tree implementation stopping a child watch will actually unwatch the parent as well, this can be reproduced with the following snippet on MacOS or Windows which uses the recursive tree implementation:
    package main
    
    import (
    	"fmt"
    	"io/ioutil"
    	"os"
    
    	"github.com/rjeczalik/notify"
    )
    
    func watch(path string) chan notify.EventInfo {
    	c := make(chan notify.EventInfo, 1)
    	if err := notify.Watch(path+"...", c, notify.All); err != nil {
    		panic(err)
    	}
    	return c
    }
    
    func main() {
    	os.MkdirAll("./a/b/c", 0775)
    	defer os.RemoveAll("./a")
    
    	// watch a child and parent path across multiple channels. This can happen in any order.
    	ch1 := watch("./a/b")
    	ch2 := watch("./a/b/c")
    
    	// unwatch ./a/b/c -- this is what triggers unwatching of ./a/b as well
    	notify.Stop(ch2)
    
    	// fire an event that will never show up because the path a/b is now unwatched
    	go func() { ioutil.WriteFile("a/b/c/d", []byte("X"), 0664) }()
    
    	// Never terminates
    	fmt.Println(<-ch1)
    }
    

    I added a new test called TestStopChild that tests explicitly for this problem.

    • The second problem was already described in #155, but is as well a problem in the recursive tree watcher implementation that does not correctly check children nodes for inactive watchpoints. I added a new test called TestAddParentAfterStop that tests explicitly for this problem. With the fixes introduced in this PR, the code snippet in #155 runs fine without any panics
    • The third problem is that on calling Stop(), RecursiveUnwatch is never called, because watchIsRecursive(nd) is called after the watchpoint was already removed and hence the check always returns that the watch is not recursive
  • Still

    Still "under active development"?

    The README for this project says it is "under active development", and positions it as "on steroids" (presumably as opposed to the more widely used fsnotify package). Is it still under active development?

  • FILE_TRUNC does not exist on Solaris

    FILE_TRUNC does not exist on Solaris

    This does not compile on Solaris due to C.FILE_TRUNC missing.

    .../go/pkg/mod/github.com/syncthing/[email protected]/watcher_fen_cgo.go:30:25: could not determine kind of name for C.FILE_TRUNC

  • build error from mac osx 10.15 fatal error: 'CoreServices/CoreServices.h' file not found

    build error from mac osx 10.15 fatal error: 'CoreServices/CoreServices.h' file not found

    hi, when I build a project which depends on notify in mac osx 10.15 , it gets below error

    the detail step is described here: would you please to check: https://github.com/ethereum/go-ethereum/issues/21835

    error info

    # github.com/rjeczalik/notify
    ../../pkg/mod/github.com/rjeczalik/[email protected]/watcher_fsevents_cgo.go:10:10: fatal error: 'CoreServices/CoreServices.h' file not found
    #include <CoreServices/CoreServices.h>
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ../../pkg/mod/github.com/rjeczalik/[email protected]/watcher_fsevents_cgo.go:10:10: note: did not find header 'CoreServices.h' in framework 'CoreServices' (loaded from '/System/Library/Frameworks')
    1 error generated.
    
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site
Bigfile -- a file transfer system that supports http, rpc and ftp protocol   https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

Dec 31, 2022
Pluggable, extensible virtual file system for Go

vfs Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as

Jan 3, 2023
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

Jan 1, 2023
Dragonfly is an intelligent P2P based image and file distribution system.
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

Jan 9, 2023
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

Jan 2, 2023
File system for GitHub
File system for GitHub

HUBFS · File System for GitHub HUBFS is a read-only file system for GitHub and Git. Git repositories and their contents are represented as regular dir

Dec 28, 2022
A virtual file system for small to medium sized datasets (MB or GB, not TB or PB). Like Docker, but for data.

AetherFS assists in the production, distribution, and replication of embedded databases and in-memory datasets. You can think of it like Docker, but f

Feb 9, 2022
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go

GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go Overview GeeseFS allows you to mount an S3 bucket as a file sys

Jan 1, 2023
Encrypted File System in Go

Getting Started: Setup the environment: Install GoLang: $ sudo apt update $ sudo apt upgrade $ sudo apt install libssl-dev gcc pkg-config $ sudo apt

Apr 30, 2022
A rudimentary go program that allows you to mount a mongo database as a FUSE file system

This is a rudimentary go program that allows you to mount a mongo database as a

Dec 29, 2021
Gokrazy mkfs: a program to create an ext4 file system on the gokrazy perm partition

gokrazy mkfs This program is intended to be run on gokrazy only, where it will c

Dec 12, 2022
Go file operations library chasing GNU APIs.
Go file operations library chasing GNU APIs.

flop flop aims to make copying files easier in Go, and is modeled after GNU cp. Most administrators and engineers interact with GNU utilities every da

Nov 10, 2022
Go library for the TOML file format

Go-toml provides the following features for using data parsed from TOML documents

Dec 27, 2022
goelftools is library written in Go for parsing ELF file.

goelftools goelftools is library written in Go for parsing ELF file. This library is inspired by pyelftools and rbelftools. Motivation The motivation

Dec 5, 2022
A FileSystem Abstraction System for Go
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Jan 9, 2023
Experimental typesetting system

ETS Experimental typesetting system This software repository contains a Lua frontend for the typesetting library “Boxes and Glue” which is an algorith

Jan 30, 2022
Go-lang based sonos standup system

Overview This is an CLI tool that can handle timed standup playback on a sonos device. It allows you to add links to audio files that will be randomly

Nov 23, 2021
Abstract File Storage

afs - abstract file storage Please refer to CHANGELOG.md if you encounter breaking changes. Motivation Introduction Usage Matchers Content modifiers S

Dec 30, 2022