The fastest file download client

Pget - The fastest file download client

.github/workflows/main.yaml codecov Go Report Card GitHub release MIT License

Description

Multi-Connection Download using parallel requests.

  • Fast
  • Resumable
  • Cross-compiled (windows, linux, macOS)

This is an example to download linux kernel. It will be finished between 15s.

pget

Disclaimer

This program comes with no warranty. You must use this program at your own risk.

Note

  • Using a large number of connections to a single URL can lead to DOS attacks.
  • The case is increasing that if you use multiple connections to 1 URL does not increase the download speed with the spread of CDNs.
    • I recommend to use multiple mirrors simultaneously for faster downloads (And the number of connections is 1 for each).

Installation

Homebrew

brew update
brew install Code-Hex/tap/pget

Go

$ go install github.com/Code-Hex/pget/cmd/pget@latest

Synopsis

This example will be used 2 connections per URL.

$ pget -p 2 MIRROR1 MIRROR2 MIRROR3

If you have created such as this file

cat list.txt
MIRROR1
MIRROR2
MIRROR3

You can do this

cat list.txt | pget -p 2

Options

  Options:
  -h,  --help                   print usage and exit
  -p,  --procs <num>            the number of connections for a single URL (default 1)
  -o,  --output <filename>      output file to <filename>
  -t,  --timeout <seconds>      timeout of checking request in seconds
  -u,  --user-agent <agent>     identify as <agent>
  -r,  --referer <referer>      identify as <referer>
  --check-update                check if there is update available
  --trace                       display detail error messages

Pget vs Wget

URL: https://mirror.internet.asn.au/pub/ubuntu/releases/21.10/ubuntu-21.10-desktop-amd64.iso

Using

time wget https://mirror.internet.asn.au/pub/ubuntu/releases/21.10/ubuntu-21.10-desktop-amd64.iso
time pget -p 6 https://mirror.internet.asn.au/pub/ubuntu/releases/21.10/ubuntu-21.10-desktop-amd64.iso

Results

wget   3.92s user 23.52s system 3% cpu 13:35.24 total
pget -p 6   10.54s user 34.52s system 25% cpu 2:56.93 total

wget 13:35.24 total, pget -p 6 2:56.93 total (6x faster)

Binary

You can download from here

Author

codehex

Owner
Kei Kamikawa
Okinawapm, Previous: Mercari, Inc.
Kei Kamikawa
Comments
  • Compatible with links that do not support range request

    Compatible with links that do not support range request

    • fixed #16
    • fixed #45
    ./pget 'https://mirrors.tuna.tsinghua.edu.cn/github-release/git-for-windows/git/Git%20for%20Windows%202.35.1%282%29/Git-2.35.1.2-32-bit.exe'
    

    it works fine

    image

  • The process cannot access the file because it is being used by another process.

    The process cannot access the file because it is being used by another process.

    PS D:\tools\pget_windows_amd64> .\pget.exe -p 6 http://www.nerc.edu.cn/wk///2016/08/02/344a06be-dc8e-4d7b-a5b9-bc1b1edb83b5/Scorm/27c69af9-30da-4f7e-8
    f71-407277c10da3.mp4 --trace
    Checking now http://www.nerc.edu.cn/wk///2016/08/02/344a06be-dc8e-4d7b-a5b9-bc1b1edb83b5/Scorm/27c69af9-30da-4f7e-8f71-407277c10da3.mp4
    Download start from http://www.nerc.edu.cn/wk///2016/08/02/344a06be-dc8e-4d7b-a5b9-bc1b1edb83b5/Scorm/27c69af9-30da-4f7e-8f71-407277c10da3.mp4
     53349059 / 53349059 [===================================================================================================================] 100.00% 6s
    
    binding with files...
     0 / 53349059 [-----------------------------------------------------------------------------------------------------------------------------]   0.00%
    Error:
    remove _27c69af9-30da-4f7e-8f71-407277c10da3.mp4.6/27c69af9-30da-4f7e-8f71-407277c10da3.mp4.6.0: The process cannot access the file because it is bein
    g used by another process.
    failed to remove a file in download location
    github.com/Code-Hex/pget.(*Data).BindwithFiles
            /Users/CodeHex/Desktop/go/src/github.com/Code-Hex/pget/util.go:251
    github.com/Code-Hex/pget.(*Pget).Run
            /Users/CodeHex/Desktop/go/src/github.com/Code-Hex/pget/pget.go:78
    main.main
            /Users/CodeHex/Desktop/go/pget/cmd/pget/main.go:13
    runtime.main
            /usr/local/opt/go/libexec/src/runtime/proc.go:188
    runtime.goexit
            /usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1998
    

    OS: Windows7 64 Pget v0.0.4, parallel file download client

  • Progress Bar freezes

    Progress Bar freezes

    The progress bar will freeze after about 5 seconds, and will not show it updating until you break (CTRL + C) and start the command again.

    Any suggestions on how to fix this?

    Thanks!

  • Support multiple URLs

    Support multiple URLs

    For example:

    pget -p 6 http://example.com/i_wnant_file.iso http://example1.com/i_wnant_file.iso http://example2.com/i_wnant_file.iso http://example3.com/i_wnant_file.iso
    

    Introduction To ensure these are the same size. check use md5 if server has md5 hash.

    The operation at this,

    untitled diagram

    I think that It is friendly to the server. but It is very difficult to make.

    Please help me. Thank you.

  • specify filename

    specify filename

    Could you add future "Specify filename"? cause, sometime any webservice return/redirect to time stamped url. (like index.html?t=20161014) and pget will fail to download this case.

    command like this: pget -O index.html http://example.com/index.html // wget like pget -f index.html http://example.com/index.html // friendly-flag? meaning File

    thanks,

  • Support retry instead of exit process

    Support retry instead of exit process

    Support retry instead of exit process

    Error:
      failed to write response body: "task[13]: \"xxxx.tar.32/xxxx.tar.32.13\"": unexpected EOF
    
  • URLにハイフンが3つ続くとエラーが出る

    URLにハイフンが3つ続くとエラーが出る

    日本語で失礼します。 youtubeの動画URLをpgetでダウンロードしてみようと思った所、下記のエラーが出ました。

    sh-3.2$ pget -p 6 "https://r6---sn-nvoxu-ioq6.googlevideo.com/videoplayback?sparams=cnr%2Cdur%2Cgcr%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm(以下略)"
    Error:
      url has not been set in argument
    

    原因を探ってみたところ、URLにハイフンが3つ続くとエラーになる模様です。

    pget -p 6 "http://r6-snnvoxuioq6.googlevideo.com"
    pget -p 6 "http://r6--snnvoxuioq6.googlevideo.com"
    

    この場合ですと、ちゃんとホストがないというエラーが出ます。

    しかし

    pget -p 6 "http://r6---snnvoxuioq6.googlevideo.com"
    

    こうなった場合だけ、Error: url has not been set in argument となってしまします。

    ご対応、よろしくお願いします。

  • Disable progress bar for scripted applications?

    Disable progress bar for scripted applications?

    I'm evaluating several download-accelerator tools, found pget, and am enjoying working with it. It would be great, however, if there were an option to disable the progress bar, as the output is otherwise difficult to understand when redirected to a logfile. (Ideally, a user could disable just the progress bar, as other basic info and statistics remain quite useful. With some tools, it's all or nothing: You get nothing at all with e.g. a --quiet switch, or you get everything, including the log-busting progress bar.)

  • Not supported range access

    Not supported range access

    With a lot files, I get this error:

    pget -p 6 <url>
    Checking now <url>
    Error:
      not supported range access: <url>
    

    By example, this happens when downloading a github repo zip.

    I think it may be related with download redirection, wget does just fine.

  • Support ignore WARNING message when process larger than 4

    Support ignore WARNING message when process larger than 4

    Support ignore this WARNING message:

    [WARNING] Using a large number of connections to 1 URL can lead to DOS attacks.
    In most cases, `4` or less is enough. In addition, the case is increasing that if you use multiple conne
    ctions to 1 URL does not increase the download speed with the spread of CDNs.
    See: https://github.com/Code-Hex/pget#disclaimer
    
    Would you execute knowing these?
     (y/n) [n]: 
    
  • [bug] does not support range request

    [bug] does not support range request

    pget 'https://mirrors.tuna.tsinghua.edu.cn/github-release/git-for-windows/git/Git%20for%20Windows%202.35.1%282%29/Git-2.35.1.2-32-bit.exe'
    
    Error:
      https://mirrors.tuna.tsinghua.edu.cn/github-release/git-for-windows/git/Git%20for%20Windows%202.35.1%282%29/Git-2.35.1.2-32-bit.exe: does not support range request
    
    

    when url not support range request , pget should not use range request and it should be compatible with this error

  • Please add support for recursive download

    Please add support for recursive download

    wget has the functionality to download files recursively from a website. but sadly it doesn't support multi-threaded and downloads one file at a time.

    Instead of wget -r ftp://example.com/test/ we want pget -p 4 -r ftp://example.com/test/

  • Error: file name too long

    Error: file name too long

    On Linux using Pget 0.1.1 installed by linuxbrew I'm getting the following error when I download a binary. How could we fix it?

    $ pget https://github.com/42wim/matterbridge/releases/download/v1.25.0/matterbridge-1.25.0-linux-64bit

    Error:
      failed to mkdir for download location: mkdir _a44ac081-f540-4721-ae1f-05c2bdd363cf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220502%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220502T142130Z&X-Amz-Expires=300&X-Amz-Signature=db8edd56f819399d6040d92ced71cba3fa7b2d7b94d5cd6255f33d7e092567f5&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=44820350&response-content-disposition=attachment%3B%20filename%3Dmatterbridge-1.25.0-linux-64bit&response-content-type=application%2Foctet-stream.1: file name too long
    
  • Self locked?

    Self locked?

    C:\vm>pget https://releases.ubuntu.com/20.04.3/ubuntu-20.04.3-live-server-amd64.iso -p 8
    [WARNING] Using a large number of connections to 1 URL can lead to DOS attacks.
    In most cases, `4` or less is enough. In addition, the case is increasing that if you use multiple connections to 1 URL does not increase the download speed with the spread of CDNs.
    See: https://github.com/Code-Hex/pget#disclaimer
    
    Would you execute knowing these?
     (y/n) [n]: y
    1.17 GiB / 1.17 GiB [---------------------------------------------------------------------------] 100.00% 54.10 MiB p/s
    
    binding with files...
    Error:
      failed to remove "_ubuntu-20.04.3-live-server-amd64.iso.8/ubuntu-20.04.3-live-server-amd64.iso.8.0" in download location: remove _ubuntu-20.04.3-live-server-amd64.iso.8/ubuntu-20.04.3-live-server-amd64.iso.8.0: The process cannot access the file because it is being used by another process.
    

    Only running one command.

Got: Simple golang package and CLI tool to download large files faster 🏃 than cURL and Wget!

Got. Simple and fast concurrent downloader. Installation ❘ CLI Usage ❘ Module Usage ❘ License Comparison Comparison in cloud server: [root@centos-nyc-

Dec 29, 2022
tmux-wormhole - download files and directories with tmux!
tmux-wormhole - download files and directories with tmux!

tmux-wormhole Use tmux and magic wormhole to get things from your remote computer to your tmux. If tmux has DISPLAY set, open the file locally! Demo U

Nov 9, 2022
📷 Command-line utility to download all photos from Instagram
📷 Command-line utility to download all photos from Instagram

Instagram Downloader This is a simple command-line tool, written in Go, to download all images from an Instagram account. Getting Started Install inst

Sep 9, 2022
A small utility command line application that can recursively download Notion pages

notionbackup A small utility command line application that can recursively download Notion pages. I needed something scriptable that could periodicall

Dec 5, 2022
📥 Command-line tool to download videos from hanime.tv

hanime Command-line tool to download videos from hanime.tv Requirements Installation Install via go get Install from source Install from release Usage

Dec 18, 2022
A golang CLI tool to download malware from a variety of sources.

mlget _____ _____ _____ _____ _____ /\ \

Jan 2, 2023
A terminal-based download manager written in Go!

goload-manager About A terminal-based download manager written in Go! This project uses grab for downloading and tcell for the terminal interface. Hea

Oct 7, 2021
This is the tool to download files from qiniu cruster manually.

This is the tool to download files from qiniu cruster manually. toCheck = []string{ sealPath, filepath.Join(cachePath, "p_aux"), filepath.Join(cachePa

Nov 25, 2021
Download and install binaries from GitHub Releases, interactively.

bget Download and install binary files from GitHub Releases. Preview Install For Mac/Linux users, you can use Homebrew to install it: brew install ego

Aug 24, 2022
Downloader written in golang to download the public data files from RUC Paraguay.

rucpy-downloader Downloader written in golang to download the public data files(RUC Paraguay) from set.gov.py. The downloader will download the public

Dec 6, 2021
CLI tool to upload object to s3-compatible storage backend and set download policy for it.
CLI tool to upload object to s3-compatible storage backend and set download policy for it.

typora-s3 CLI tool to upload object to s3-compatible storage backend and set download policy for it. Build $ git clone https://github.com/fengxsong/ty

Dec 29, 2021
Little golang app that allows you to download a youtube video as mp3, and optionally embed ID3 tags -Cover Art, Artist ...-

yt2mp3 Little golang app that allows you to download a youtube video as mp3, and optionally embed ID3 tags -Cover Art, Artist ...- Instructions At the

Dec 25, 2021
Mass download all github repositories(public & private) of an organization, ideally in a few seconds.

Git Mass Mass download all github repositories(public & private) of an organization, ideally in a few seconds. Writing this as a simple bash script wo

Dec 27, 2021
Command-line program to download videos from YouTube.com and other video sites

youtube-dl - download videos from youtube.com or other video platforms INSTALLATION DESCRIPTION OPTIONS CONFIGURATION OUTPUT TEMPLATE FORMAT SELECTION

Jan 9, 2023
You-Get is a tiny command-line utility to download media contents (videos, audios, images) from the Web,
You-Get is a tiny command-line utility to download media contents (videos, audios, images) from the Web,

You-Get NOTICE: Read this if you are looking for the conventional "Issues" tab. You-Get is a tiny command-line utility to download media contents (vid

Jan 2, 2023
An easy to use go program to download videos from Reddit

Gedditsave Download videos from reddit from the command line. Use guide geddit -

Oct 17, 2022
An easy to use go program to download videos from Reddit.
An easy to use go program to download videos from Reddit.

Gettit Download videos from reddit from the command line. Use guide gettit -u post_url Example : gettit -u https://www.reddit.com/r/programminghorror/

Dec 30, 2022
Gowget - A simple wget tool built using golang to download files from the internet whether it is zip

gowget wget built using golang About: A simple wget tool built using golang to d

May 21, 2022