pcp - πŸ“¦ Command line peer-to-peer data transfer tool based on libp2p.

pcp - Peer Copy

standard-readme compliant Go Report Card Maintainability Github Releases Download Count

Command line peer-to-peer data transfer tool based on libp2p.

Demo animation

Table of Contents

Motivation

There already exists a long list of file transfer tools (see Related Efforts), so why bother building another one? The problem I had with the existing tools is that they rely on a limited set of servers to orchestrate peer matching and data relaying which poses a centralisation concern. Many of the usual centralisation vs. decentralisation arguments apply here, e.g. the servers are single points of failures, the service operator has the power over whom to serve and whom not, etc. Further, as this recent issue in croc shows, this is a real risk for sustainable operation of the provided service.

Project Status

The tool is in a very early stage, and I'm aware of performance, usability and security issues. Don't use it for anything serious. Although I criticised tools like magic-wormhole or croc above, they are amazing and way more mature.

There are also drawbacks with this approach: It's slower than established centralised methods if you want to transmit data across network boundaries. A DHT query to find your peer can easily take several minutes. Further, the bandwidth and geographic location of a potential relaying peer is not guaranteed which can lead to long transmission times.

How does it work?

When running pcp send you'll see four random words from a list of the Bitcoin improvement proposal BIP39. There are lists in nine different languages of 2048 words each, currently only english is supported. The first word is interpreted as a channel ID in the range from 0 to 2047. pcp advertises the identifier /pcp/{unix-timestamp}/{channel-id} in its local network via mDNS and a hashed version of this string in the DHT of IPFS. The unix timestamp is the current time truncated to 5 minutes and the prefix /pcp is the protocol prefix. In the future: When you enter a new 5-minute interval while pcp send is running it advertises an updated identifier.

To receive the file your peer enters pcp receive four-words-from-above and pcp uses the first word together with the current time truncated to 5 minutes to find the sending peer in the DHT and in your local network via mDNS. It also searches for an identifier of the previous 5-minute interval. As soon as the peer is found, both do a password authenticated key exchange (PAKE) to authenticate each other. In this procedure a comparably weak password (four-words-from-above) gets replaced with a strong session key that is used to encrypt all future communication. The default TLS encryption that libp2p provides is not sufficient in this case as we could still, in theory, talk to a wrong peer - just encrypted.

After the peer is authenticated the receiver must confirm the file transfer, and the file gets transmitted.

Usage

The sending peer runs:

$ pcp send my_file
Code is:  bubble-enemy-result-increase
On the other machine run:
	pcp receive bubble-enemy-result-increase

The receiving peer runs:

$ pcp receive bubble-enemy-result-increase
Looking for peer bubble-enemy-result-increase...

If you're on different networks the lookup can take quite long (~ 2-3 minutes). Currently, there is no output while both parties are working on peer discovery, so just be very patient.

Install

Release download

Head over to the releases and download the latest binary for your platform.

From source

To compile it yourself clone the repository:

git clone https://github.com/dennis-tra/pcp.git

Navigate into the pcp folder and run:

go install cmd/pcp/pcp.go # Go 1.13 or higher is required

Make sure the $GOPATH/bin is in your PATH variable to access the installed pcp executable.

Package managers

It's on the roadmap to distribute pcp via apt, yum, brew, scoop and more ...

Development

Protobuf definitions

First install the protoc compiler:

make tools # downloads gofumpt and protoc
make proto # generates protobuf

The current proto definitions were generated with libprotoc 3.14.0.

Feature Roadmap

Shamelessly copied from croc:

  • allows any two computers to transfer data (using a relay)
  • provides end-to-end encryption (using PAKE)
    • βœ… yup, it uses pake/v2 from croc
  • enables easy cross-platform transfers (Windows, Linux, Mac)
    • πŸ€” βœ… only tested Linux <-> Mac
  • allows multiple file transfers
    • ❌ not yet
  • allows resuming transfers that are interrupted
    • ❌ not yet
  • local server or port-forwarding not needed
  • ipv6-first with ipv4 fallback
    • πŸ€” I think that's the case, but I'm not sure about the libp2p internals
  • can use proxy, like tor
    • ❌ not yet

Related Efforts

  • croc - Easily and securely send things from one computer to another
  • magic-wormhole - get things from one computer to another, safely
  • dcp - Remote file copy, powered by the Dat protocol.
  • iwant - CLI based decentralized peer to peer file sharing
  • p2pcopy - Small command line application to do p2p file copy behind firewalls without a central server.
  • zget - Filename based peer to peer file transfer
  • sharedrop - Easy P2P file transfer powered by WebRTC - inspired by Apple AirDrop
  • filepizza - Peer-to-peer file transfers in your browser
  • toss - Dead simple LAN file transfers from the command line
  • Forgot yours? Open an issue or submit a PR :)

Maintainers

@dennis-tra.

Acknowledgments

  • go-libp2p - The Go implementation of the libp2p Networking Stack.
  • pake/v2 - PAKE library for generating a strong secret between parties over an insecure channel
  • progressbar - A really basic thread-safe progress bar for Golang applications

Contributing

Feel free to dive in! Open an issue or submit PRs.

Other Projects

You may be interested in one of my other projects:

  • image-stego - A novel way to image manipulation detection. Steganography-based image integrity - Merkle tree nodes embedded into image chunks so that each chunk's integrity can be verified on its own.

License

Apache License Version 2.0 Β© Dennis Trautwein

Owner
Dennis Trautwein
BSc./MSc. in extraterrestrial/solid state physics | iOS/WebDev WellingtonNZ/MelbourneAU | [email protected] HamburgGER | Dev@OriginStamp CH
Dennis Trautwein
Comments
  • Bump github.com/libp2p/go-libp2p from 0.13.0 to 0.18.1

    Bump github.com/libp2p/go-libp2p from 0.13.0 to 0.18.1

    Bumps github.com/libp2p/go-libp2p from 0.13.0 to 0.18.1.

    Release notes

    Sourced from github.com/libp2p/go-libp2p's releases.

    v0.18.0

    We're happy to announce the v0.18.0 release of go-libp2p.

    πŸ›  Breaking Changes

    In this release, we're continuing our repo consolidation. We've moved the following repositories into go-libp2p:

    • go-libp2p-autonat
    • go-libp2p-discovery
    • go-libp2p-connmgr
    • go-addr-utils
    • go-libp2p-netutil
    • go-sockaddr

    If you've been using functions from these repositories, please make sure to point those to go-libp2p.

    πŸ”¦ Highlights

    The main feature of v0.18.0 is the Resource Manager. The Resource Manager allows the configuration of (among others):

    • the number of incoming / outgoing connections
    • the number of incoming / outgoing streams
    • the number of streams per protocol, and per service
    • memory usage

    For details on how this works, and how to change limits, refer to the README of the Resource Manager.

    In this release, we've finished the transition from ipfs/go-log@v1 to ipfs/go-log@v2. One less dependency!

    Changelog

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump github.com/libp2p/go-libp2p from 0.13.0 to 0.18.0

    Bump github.com/libp2p/go-libp2p from 0.13.0 to 0.18.0

    Bumps github.com/libp2p/go-libp2p from 0.13.0 to 0.18.0.

    Release notes

    Sourced from github.com/libp2p/go-libp2p's releases.

    v0.18.0

    We're happy to announce the v0.18.0 release of go-libp2p.

    πŸ›  Breaking Changes

    In this release, we're continuing our repo consolidation. We've moved the following repositories into go-libp2p:

    • go-libp2p-autonat
    • go-libp2p-discovery
    • go-libp2p-connmgr
    • go-addr-utils
    • go-libp2p-netutil
    • go-sockaddr

    If you've been using functions from these repositories, please make sure to point those to go-libp2p.

    πŸ”¦ Highlights

    The main feature of v0.18.0 is the Resource Manager. The Resource Manager allows the configuration of (among others):

    • the number of incoming / outgoing connections
    • the number of incoming / outgoing streams
    • the number of streams per protocol, and per service
    • memory usage

    For details on how this works, and how to change limits, refer to the README of the Resource Manager.

    In this release, we've finished the transition from ipfs/go-log@v1 to ipfs/go-log@v2. One less dependency!

    Changelog

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • could not establish enough connections to bootstrap peers; BUT I can ping them

    could not establish enough connections to bootstrap peers; BUT I can ping them

    when I run pcp send command, the output is like :

    could not establish enough connections to bootstrap peers
    	failed to dial QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ: all dials failed
      * [/ip4/104.131.131.82/tcp/4001] dial tcp4 0.0.0.0:38687->104.131.131.82:4001: i/o timeout
    	failed to dial QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb: all dials failed
      * [/ip6/2604:1380:2000:7a00::1/tcp/4001] dial tcp6 [2604:1380:2000:7a00::1]:4001: connect: network is unreachable
      * [/ip4/147.75.83.83/tcp/4001] dial tcp4 0.0.0.0:38687->147.75.83.83:4001: i/o timeout
    	failed to dial QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN: all dials failed
      * [/ip6/2604:1380:1000:6000::1/tcp/4001] dial tcp6 [2604:1380:1000:6000::1]:4001: connect: network is unreachable
      * [/ip4/147.75.109.213/tcp/4001] dial tcp4 0.0.0.0:38687->147.75.109.213:4001: i/o timeout
    this means you will only be able to transfer files in your local network
    
    
    

    But I can ping these IP Address ping 104.131.131.82, ping 147.75.109.213 Which means the network between me and the bootstrap peers is reachable.

    PING 147.75.109.213 (147.75.109.213) type `Ctrl-C` to abort
     seq=0 32bytes from 147.75.109.213: ttl=46 time=217.729074ms
     seq=1 32bytes from 147.75.109.213: ttl=46 time=217.792354ms
     seq=2 32bytes from 147.75.109.213: ttl=46 time=217.940214ms
     seq=3 32bytes from 147.75.109.213: ttl=46 time=217.733247ms
    ───────── 147.75.109.213 ping statistics ─────────
    PACKET STATISTICS: 4 transmitted => 4 received (0% loss)
    ROUND TRIP: min=217.729074ms avg=217.798722ms max=217.940214ms stddev=85.437Β΅s
    

    So is this issue caused by my network or the network between me and the bootstrap peer or the DHT?

    I really want to try this project, it looks fun and inspiring

  • Enhance hole punching

    Enhance hole punching

    We can try to establish direct connection before transfer (while hole-punching is going on background). I also enable libp2p "p2p-holepunching" log when -debug flag set, so we can see why hole punching fail (in my case it will mostly timeout with the default dial timeout value).

  • brew package conflict with PCP - Perfomance CoPilot

    brew package conflict with PCP - Perfomance CoPilot

    the package entry in brew conflicts with Performance CoPilot.

    See https://github.com/performancecopilot/pcp/blob/main/INSTALL.md "section 2. Mac OS X Installation" for more information.

transfer.sh - Easy and fast file sharing from the command-line.

Easy and fast file sharing from the command-line. This code contains the server with everything you need to create your own instance.

Jan 2, 2023
A web based drag and drop file transfer tool for sending files across the internet.

DnD A file transfer tool. Demo Usage Get go get github.com/0xcaff/dnd or download the latest release (you don't need go to run it) Run dnd Now navig

Dec 16, 2022
Personal tool used for transfer files between servers.

transfer Personal tool used for transfer files between servers. Usage: transfer [global options] <verb> [verb options] Global options: -h, --

Jan 5, 2022
mini file transfer tool, use it just curl o wget

miniTransfer mini file transfer tool, use it just curl o wget How to use upload file curl -T localFileName 127.0.0.1:1234 # default save as localFileN

Jan 12, 2022
oDrop, a fast efficient cross-platform file transfer software for server and home environments

oDrop is a cross-platform LAN file transfer software to efficiently transfer files between computers, oDrop is useful in environments where GUI is not available.

Jun 4, 2022
Yet another netcat for fast file transfer

nyan Yet another netcat for fast file transfer When I need to transfer a file in safe environment (e.g. LAN / VMs), I just want to use a simple comman

Apr 30, 2022
Transfer your local .ENV files for another developer

Zip/Unzip local .env file Transfer your local .ENV files for another developer. for Zip: run in root where .env files ./ziper_local_env created "resu

Oct 20, 2021
Built in user interface, LAN file transfer, such as mobile phone, computer, tablet, different operating system
Built in user interface, LAN file transfer, such as mobile phone, computer, tablet, different operating system

Modao Built in user interface, LAN file transfer, such as mobile phone, computer, tablet, different operating systems, etc., as well as text transfer

May 7, 2022
Transfer - A simple go application for uploading, downloading and checksumming of files

Transfer This project holds a simple go application for uploading, downloading a

Aug 18, 2022
FileTransferGo - File Transfer With Golang

FileTransferGo Packages used ?? Go: Gin (http server) ?? Cobra (CLI command fram

Jun 24, 2022
Simple temporary file upload and transfer web application coding with Go language.

Temp File Transfer Web Application Simple temporary file upload and transfer web application coding with Go language. Explore the Golang Β» Live Demo T

Dec 2, 2022
πŸ–₯ Securely transfer and send anything between computers with TUI.
πŸ–₯ Securely transfer and send anything between computers with TUI.

??️ Securely transfer and send anything between computers with TUI. Installation ways

Dec 21, 2022
Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations

Filestore Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations (store named binar

Nov 7, 2021
Easy and fast file sharing from the command-line.

Easy and fast file sharing from the command-line. This code contains the server with everything you need to create your own instance.

Jan 2, 2023
Golang PoC software for reliable file transfers over a data diode. DIY gigabit data diode hardware instructions
Golang PoC software for reliable file transfers over a data diode. DIY gigabit data diode hardware instructions

DIY Data Diode Simple DIY gigabit data diode (hardware and software). Presented at SEC-T 2021. Hardware By doing a simple hardware mod to a fiber conv

Dec 1, 2022
proxyd proxies data between TCP, TLS, and unix sockets

proxyd proxyd proxies data between TCP, TLS, and unix sockets TLS termination: Connecting to a remote application's unix socket: +---------+

Nov 9, 2022
An experimental project to build a framework for naming and sharing files and other data securely
An experimental project to build a framework for naming and sharing files and other data securely

Upspin Documentation: upspin.io About the project Upspin is an experimental project to build a framework for naming and sharing files and other data s

Oct 20, 2021
✨Super-fast and easy data transforming with any interfaces

BELT Super-fast and easy data transforming with any interfaces Just connect Belt to your Factory Methods Transform data Map Filter Assistance Pipe App

Feb 13, 2022
Peerster refers to a gossip-based P2P system composed of multiple peers interacting with each other

Peerster design Peerster refers to a gossip-based P2P system composed of multiple peers interacting with each other. A peer refers to an autonomous en

Jan 22, 2022