Pure-Go library for cross-platform local peer discovery using UDP multicast :woman: :repeat: :woman:

peerdiscovery

Code coverage Go Report Go Doc

Pure-go library for cross-platform thread-safe local peer discovery using UDP multicast. I needed to use peer discovery for croc and everything I tried had problems, so I made another one.

Install

Make sure you have Go 1.5+.

go get -u github.com/schollz/peerdiscovery

Usage

The following is a code to find the first peer on the local network and print it out.

discoveries, _ := peerdiscovery.Discover(peerdiscovery.Settings{Limit: 1})
for _, d := range discoveries {
    fmt.Printf("discovered '%s'\n", d.Address)
}

Here's the output when running on two computers. (Run these gifs in sync by hitting Ctl + F5).

Computer 1:

computer 1

Computer 2:

computer 1

For more examples, see the scanning examples (ipv4 and ipv6) or the docs.

Testing

To test the peer discovery with just one host, one can launch multiple containers. The provided Dockerfile will run the example code. Please make sure to enable Docker's IPv6 support if you are using IPv6 for peer discovery.

# Build the container, named peertest
$ docker build -t peertest .

# Execute the following command in multiple terminals
$ docker run -t --rm peertest
Scanning for 10 seconds to find LAN peers
 100% |████████████████████████████████████████|  [9s:0s]Found 1 other computers
0) '172.17.0.2' with payload 'zqrecHipCO'

Contributing

Pull requests are welcome. Feel free to...

  • Revise documentation
  • Add new features
  • Fix bugs
  • Suggest improvements

Thanks

Thanks @geistesk for adding IPv6 support and a Notify func, and helping maintain! Thanks @Kunde21 for providing a bug fix and massively refactoring the code in a much better way. Thanks @robpre for finding and fixing bugs. Thanks @shvydky for adding dynamic payloads.

License

MIT

Comments
  • Ignoring configured address family can lead to deadlock on MacOS

    Ignoring configured address family can lead to deadlock on MacOS

    After getting the list of interfaces, interfaces not having at least one address of the configured address family are not skipped.

    Sending a multicast with an IPv4 connection on an IPv6-only interface results (at least on MacOS) in a deadlock.

    Additionally, you should check the interface flags if is up and if it supports multicast at all.

    This issue might be the root cause of https://github.com/schollz/croc/issues/313

  • Discovery hangs if executed in a loop

    Discovery hangs if executed in a loop

    I tried executing peerdiscovery.Discover() function in a loop and it stucks randomly after some time. After a little debugging, my hunch is, this is due to connection not being closed in listen() function. Please let me know if this is the actual reason or it is because of something else.

  • can't find each other

    can't find each other

    discoveries, _ := peerdiscovery.Discover(peerdiscovery.Settings{Limit: -1,TimeLimit: -1})
    for _, d := range discoveries {
        fmt.Printf("discovered '%s'\n", d.Address)
    }
    

    run on a, b and c that are independent mac. result: a can discover b and c b can discover c c can't discover a or b

    go 1.14 mac 10.15.3

  • IPv6 support

    IPv6 support

    This pull request adds IPv6 multicast scans by introducing the IPVersion field to the Settings struct, defaulting to IPv4 to not break the compatibility. However, settings IPVersion: IPv6 will execute the same peer discovery service, just for IPv6 instead.

    The code is a bit ugly at some points, because both ipv4.PacketConn and ipv6.PacketConn does not implement net.PacketConn. Therefore I treated them as an empty interface.

    Please tell me what you think and thanks a lot for this library!


    Random note regarding testing: I wrote a small Dockerfile which launches - after enabling IPv6 - some containers for testing.

    $ cat Dockerfile
    FROM golang:1.11
    
    WORKDIR /peerdiscovery
    COPY . .
    RUN go build ./examples/main.go
    
    CMD ["/peerdiscovery/main"]
    
    $ docker build -t peertest .
    . . .
    
    # run the following command in multiple terminals
    $ docker run -t --rm peertest
    
  • Can't compile using gcc-go

    Can't compile using gcc-go

    I've tested 2 gcc-go versions on Arch Linux 10.2.0 and 10.3.0 go version go1.14.4 gccgo (GCC) 10.2.0 linux/amd64 go version go1.14.6 gccgo (GCC) 10.3.0 linux/amd64

    I've also tested on Debian testing, just to see if it was maybe Arch issue, but got same error gccgo-go on Debian testing reports go version go1.14.6 gccgo (Debian 10.2.1-6) 10.2.1 20210110 linux/amd64 the error is

    go get -u github.com/schollz/peerdiscovery
    # github.com/schollz/peerdiscovery
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:381:6: error: incompatible type for receiver (cannot use type PacketConn4 as type socket.Conn)
      381 | type PacketConn4 struct {
          |      ^
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:381:6: error: incompatible type for receiver (cannot use type PacketConn4 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:381:6: error: incompatible type for receiver (cannot use type PacketConn4 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:381:6: error: incompatible type for receiver (cannot use type PacketConn4 as type ipv4.genericOpt)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:381:6: error: incompatible type for receiver (cannot use type PacketConn4 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:381:6: error: incompatible type for receiver (cannot use type PacketConn4 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:381:6: error: incompatible type for receiver (cannot use type PacketConn4 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:381:6: error: incompatible type for receiver (cannot use type PacketConn4 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:381:6: error: incompatible type for receiver (cannot use type PacketConn4 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:396:6: error: incompatible type for receiver (cannot use type PacketConn6 as type socket.Conn)
      396 | type PacketConn6 struct {
          |      ^
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:396:6: error: incompatible type for receiver (cannot use type PacketConn6 as type ipv6.genericOpt)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:396:6: error: incompatible type for receiver (cannot use type PacketConn6 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:396:6: error: incompatible type for receiver (cannot use type PacketConn6 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:396:6: error: incompatible type for receiver (cannot use type PacketConn6 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:396:6: error: incompatible type for receiver (cannot use type PacketConn6 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:396:6: error: incompatible type for receiver (cannot use type PacketConn6 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:396:6: error: incompatible type for receiver (cannot use type PacketConn6 as type socket.Conn)
    ../../go/src/github.com/schollz/peerdiscovery/peerdiscovery.go:396:6: error: incompatible type for receiver (cannot use type PacketConn6 as type socket.Conn)
    

    tagging https://github.com/schollz/croc/issues/360

  • Fix StopChan behaviour

    Fix StopChan behaviour

    This seems to fix the exit behaviour, since the exit variable was created for the for loop scope, setting it to true at the end of the loop meant it wasn't being caught by the check on line 214.

    I wasn't sure if we should just move the if to under the select, but this seemed like the lowest-impact fix.

    Thanks for this module! :)

  • Enable unlimit scanning and notify function

    Enable unlimit scanning and notify function

    Enabling unlimitted scanning by setting TimeLimit to a negative value and setting Notify to a function, consuming a Discovered struct, will let the peerdiscovery library run forever. Each discovery, regardless of wether known before or not, will be passed to the Notify function. This implementation is oriented by the observer pattern.

    I needed the described change of the library for my use case. This enables scanning for new peers in the background and letting me decide if I want to establish a connection to those hosts.

    Please give me a feedback, if this is helpful for you or this library at all?

  • Various improvements

    Various improvements

    • Restructure the library: the library was split into several files to make keeping only the necessary context a bit easier
    • GC for lost peers: for long-running apps it's useful to have some way to know if we didn't see a peer for a while
    • Introduce support for peer metadata + Lost peer notifications: useful if you want to store some information relevant to this specific peer to perform something upon loosing the peer
    • Shutdown support: simple way to stop the listener
    • Fix a few error shadowing issues

    I made my best effort to keep the backwards compatibility, and as far as I know this PR should be backward-compatible.

  • Reintroduce Loopback Device Support

    Reintroduce Loopback Device Support

    The filterInterfaces function reduces the useable network interfaces for peer discovery. Due to two changes introduced in v1.6.4^0 and v1.6.5^1, a self-discovery over a loopback device was no longer possible.

    For reasons, a device either holds the loopback flag or the multicast flag, not both together. This commit reflects this.

    Furthermore, some refactoring of the filterInterfaces function was performed to ease its understandability.

  • Close PacketConn listener when StopChan signals exit

    Close PacketConn listener when StopChan signals exit

    One way to address https://github.com/schollz/peerdiscovery/issues/28

    While not perfect it helps for me, where a long running process repeatably start/stops discovering. Else the process will stop at some point with "too many open files".

  • Listen socket keeps opened indefinitely when stopped via StopChan

    Listen socket keeps opened indefinitely when stopped via StopChan

    When using StopChan to stop the discovery the listen() function never returns and therefore keeps the socket opened indefinitely.

    I was able to quickly hack it in a way so that the local exit variable was part of the peerDiscovery type and listen() would check if that variable is set and quits.

    Not sure if that's nice though. Also, in case no more peer discoveries are happening that function would still sit there forever with its socket listening for valid data.

  • Make sure listening Go routine exits when done

    Make sure listening Go routine exits when done

    A proposal. Use a deadline for the read operation in the Go routine. In case the routine is supposed to shut down but no traffic is happening on the multicast address this routine never gets the chance to quit.

    Instead time out the read operation after 5 seconds of no data and check whether shut down is expected.

    ~~Note that this is done against master branch. In case #29 gets accepted in some form the check for shutdown should be extended here too.~~

  • Doesn't work with AllowSelf on windows

    Doesn't work with AllowSelf on windows

    I've tried the IPV4 example with AllowSelf set to true on windows, without any other changes but It doesn't discover other local peers.

    But it works on my android 10 device android/arm64 cross-compiled on windows run via termux. Windows can't discover the android peer but android can discover another instance of itself and also windows (in the same wifi).

  • DisableBroadcast property could be dynamic

    DisableBroadcast property could be dynamic

    Hi, We started using the framework to quickly span multiple nodes for some MPC tests, and we really really like it. It allows us to mostly forget about the network setup. Great work! What would be nice is a little of dynamic management, especially Broadcast and Interval. Or, alternatively, a slightly better restart capability (some go routines like listen() are tricky ). Would you accept such a PR for a review? Cheers!

  • cant discover others when wake from hibernate

    cant discover others when wake from hibernate

    how to fix it? cant discover others when a win10 host waked from hibernate, others can discover it. if reboot or switch the lan of win10, then all gone well.

    go 1.14 mac 10.15.3 win10 professional

  • Memory leak

    Memory leak

    I have been using peerdiscovery in an app for work and it is fantastic. It makes finding other machines and making a distributed cluster quick and easy. Good work!!! I have noticed that when using it in a loop for continuous discovery and building/rebuilding a client list, the application memory usage was increasing. Using the example and running pprof it seems that there is a memory leak with the for loop which scans the socket made. This is running in goroutine which from what I can see doesn't exit.

    Type: inuse_space
    Time: Feb 10, 2020 at 7:40pm (EST)
    Entering interactive mode (type "help" for commands, "o" for options)
    (pprof) top20
    Showing nodes accounting for 1.61MB, 100% of 1.61MB total
          flat  flat%   sum%        cum   cum%
        1.61MB   100%   100%     1.61MB   100%  github.com/schollz/peerdiscovery.(*peerDiscovery).listen
    (pprof) list listen
    Total: 1.61MB
    ROUTINE ======================== github.com/schollz/peerdiscovery.(*peerDiscovery).listen in D:\goprogs\src\github.com\schollz\peerdiscovery\peerdiscovery.go
        1.61MB     1.61MB (flat, cum)   100% of Total
             .          .    322:           }
             .          .    323:   }
             .          .    324:
             .          .    325:   // Loop forever reading from the socket
             .          .    326:   for {
        1.61MB     1.61MB    327:           buffer := make([]byte, maxDatagramSize)
             .          .    328:           var (
             .          .    329:                   n       int
             .          .    330:                   src     net.Addr
             .          .    331:                   errRead error
             .          .    332:           )
    (pprof)`
    
    The above heap profile doesn't show a great deal of memory usage, however over the long run it all adds up and I had it crash the app due to memory restrictions. I resolved this by adding a bool variable into the peerdiscovery struct which I set to true at the end of the Discover method. In the socket for loop I run a check of the bool variable and break the loop if true.
    
    Crude solution but maybe something to consider if making more updates
Zero Trust Network Communication Sentinel provides peer-to-peer, multi-protocol, automatic networking, cross-CDN and other features for network communication.
Zero Trust Network Communication Sentinel provides peer-to-peer, multi-protocol, automatic networking, cross-CDN and other features for network communication.

Thank you for your interest in ZASentinel ZASentinel helps organizations improve information security by providing a better and simpler way to protect

Nov 1, 2022
An imageboard, but images are stored in a peer-to-peer network
An imageboard, but images are stored in a peer-to-peer network

Interplanetary File Dumpster An imageboard, but images are stored in a peer-to-peer network Features: Easy file sharing without registration and SMS.

Sep 30, 2022
DeepValueNetwork is a peer-to-peer database network managed and hosted by its community.

DeepValueNetwork To understand what DeepValueNetwork will be, I suggest you read this document. In progress This software is currently being developed

Dec 10, 2022
📦 Command line peer-to-peer data transfer tool based on libp2p.

pcp - Peer Copy Command line peer-to-peer data transfer tool based on libp2p. Table of Contents Motivation Project Status How does it work? Usage Inst

Jan 5, 2023
Group peer to peer video calls for everyone written in Go and TypeScript

Peer Calls v4 WebRTC peer to peer calls for everyone. See it live in action at peercalls.com. The server has been completely rewriten in Go and all th

Dec 30, 2022
Peer-to-peer hypermedia protocol
Peer-to-peer hypermedia protocol

IPFS powers the Distributed Web A peer-to-peer hypermedia protocol to make the web faster, safer, and more open. TL;DR Get help and talk about ideas i

Jan 5, 2023
Steve - A peer-to-peer (p2p) decentralized network

Steve Steve is a peer-to-peer (p2p) decentralized network that enables people to

Feb 5, 2022
UDP Transport: compress, encrypt and send any data reliably over unreliable UDP connections

udpt UDP Transport Compresses, encrypts and transfers data between a sender and receiver using UDP protocol. Features and Design Aims: Avoid the overh

Nov 5, 2022
UDP output for beats to send events over UDP.

beats-udp-output How To Use Clone this project to elastic/beats/libbeat/output/ Modify elastic/beats/libbeat/publisher/includes/includes.go : // add i

Dec 11, 2021
Cross platform local network clipboard sync.

clipSync Synchronize clipboard content across multiple devices. Features Rapidly synchronize clipboard content to all devices in same local network. S

Jan 3, 2022
Cross-platform, multi-server chat app built using Fyne
Cross-platform, multi-server chat app built using Fyne

Fibro Really early days proof of concept for a cross-platform multi-server chat. Feel free to play, but don't expect it to do much! Features Multiple

Sep 10, 2022
A toy MMO example built using Ebiten and WebRTC DataChannels (UDP)
A toy MMO example built using Ebiten and WebRTC DataChannels (UDP)

Ebiten WebRTC Toy MMO ⚠️ This is a piece of incomplete hobby work and not robust. Please read the "Why does this project exist?" section. What is this

Aug 28, 2022
P2P Forwarder - a tool for farwarding tcp/udp ports. Made using libp2p.
P2P Forwarder - a tool for farwarding tcp/udp ports. Made using libp2p.

P2P Forwarder A tool for farwarding ports. Made using libp2p. How it works A: opens desired ports ports inside P2P Forwarder A: shares it's id from P2

Nov 14, 2022
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites by using passive online sources
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites by using passive online sources

Subfinder is a subdomain discovery tool that discovers valid subdomains for websites. Designed as a passive framework to be useful for bug bounties and safe for penetration testing.

Jan 1, 2023
Cross-poster - A cross-posting tool for golang
Cross-poster - A cross-posting tool for golang

How to start cp config.example.json config.json update config in config.json Bui

Feb 19, 2022
cross-platform library for sending desktop notifications

Golang-Toast cross-platform library for sending desktop notifications Installation go get

Nov 24, 2022
A Crypto-Secure, Production-Grade Reliable-UDP Library for golang with FEC
 A Crypto-Secure, Production-Grade Reliable-UDP Library for golang with FEC

Introduction kcp-go is a Production-Grade Reliable-UDP library for golang. This library intents to provide a smooth, resilient, ordered, error-checked

Dec 28, 2022
SOCKS Protocol Version 5 Library in Go. Full TCP/UDP and IPv4/IPv6 support

socks5 中文 SOCKS Protocol Version 5 Library. Full TCP/UDP and IPv4/IPv6 support. Goals: KISS, less is more, small API, code is like the original protoc

Jan 8, 2023