Blazer - Concurrent file downloader

Blazer - Concurrent file downloader

  • Control thread count
  • Resume from interruption
  • File integrity check - SHA256

Usage

blazer -url=example.com/1.pdf -t=10

Flags

blazer -h
Usage of blazer:
  -checksum string
    	checksum SHA256(currently supported) to verify file
  -out string
    	output path to store the downloaded file
  -t int
    	Thread count - Number of concurrent downloads (default 10)
  -url string
    	Valid URL to download
  -v	prints current version of blazer

Benchmarks

Name Size Blazer cURL Wget
Debian ISO 300 MB 1min 10 sec (25 threads) 2min 40 sec 3 min 10 sec
Windows-10 5.4 GB 20min 52sec (25 threads) 46min 52 sec 40 min 25 sec

How file resumption work ?

If download is either interrupted manually or network error (Timeout, RCP - happens if "-t" is a large number than server can handle) then those segments may fail and you may have to restart the download with same URL and thread count for retry to work because, temp folder name is a hash of URL and thread count.

Demo

asciicast

Install

or

  • Build from source:
git clone [email protected]:arvyshka/blazer.git
make package
Owner
Арв
Love Golang, C++, Distributed systems, High load services
Арв
Comments
  • issue-25: pre check range support on server

    issue-25: pre check range support on server

    Issue link: https://github.com/arvyshka/blazer/issues/25

    • Checked the "Accept-Ranges" header to check if the server supports the Range header in the request.
    • Created String constant for user message
  •  Don't merge files if the download is incomplete & code cleanup #43

    Don't merge files if the download is incomplete & code cleanup #43

    • Remove global variables and pass as dependency
    • Fixed: https://github.com/arvyshka/blazer/issues/30
    • Fixed: https://github.com/arvyshka/blazer/issues/42
  • Improve code quality

    Improve code quality

    • Apply clean code
    • Add comments for exported functions
    • Restructure the application to comply according to go standards
    • Имена пакетов и функций неидиоматичны (пакет "data", к примеру).
  • Don't merge files if the download is incomplete & code cleanup

    Don't merge files if the download is incomplete & code cleanup

    • Remove global variables and pass as dependency
    • Fixed: https://github.com/arvyshka/blazer/issues/30
    • Fixed: https://github.com/arvyshka/blazer/issues/42
  • Do not merge the files if segment download is incomplete

    Do not merge the files if segment download is incomplete

    before merging the file we are not checking if all the segments were downloaded or not. And also moved the segment merging out of the network code for more clarity.

  • Publish as a homebrew package

    Publish as a homebrew package

    For homebrew users like me it would be very cool if you can publish blazer as a package so that we can do

    brew install blazer
    

    and have it installed on our systems. Homebrew is majorly used for package management by mac users. It is also available for linux as well. :-)

  • Download breaks (existing issue, unrelated to the PR of Range header)

    Download breaks (existing issue, unrelated to the PR of Range header)

    blazer -url=https://speed.hetzner.de/100MB.bin t=1 Fetching file meta.. Can't initiate download Head "https://speed.hetzner.de/100MB.bin": EOF

  • Checksum doesn't work if output file is set

    Checksum doesn't work if output file is set

    blazer -url=https://schooleverywhere-elquds.com/lib/uploadbook/179121479125786First_Children__s_Encyclopedia.pdf -t=200 -out=test.pdf -checksum=ea008f32a265d322bb3956afbd7c39d1b010b28dfc659c7b7ad3ed20bc0de339

Related tags
Abstract File Storage

afs - abstract file storage Please refer to CHANGELOG.md if you encounter breaking changes. Motivation Introduction Usage Matchers Content modifiers S

Dec 30, 2022
a tool for handling file uploads simple

baraka a tool for handling file uploads for http servers makes it easier to make operations with files from the http request. Contents Install Simple

Nov 30, 2022
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site
Bigfile -- a file transfer system that supports http, rpc and ftp protocol   https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

Dec 31, 2022
Go file operations library chasing GNU APIs.
Go file operations library chasing GNU APIs.

flop flop aims to make copying files easier in Go, and is modeled after GNU cp. Most administrators and engineers interact with GNU utilities every da

Nov 10, 2022
Read csv file from go using tags

go-csv-tag Read csv file from Go using tags The project is in maintenance mode. It is kept compatible with changes in the Go ecosystem but no new feat

Nov 16, 2022
File system event notification library on steroids.

notify Filesystem event notification library on steroids. (under active development) Documentation godoc.org/github.com/rjeczalik/notify Installation

Dec 31, 2022
Pluggable, extensible virtual file system for Go

vfs Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as

Jan 3, 2023
An epoll(7)-based file-descriptor multiplexer.

poller Package poller is a file-descriptor multiplexer. Download: go get github.com/npat-efault/poller Package poller is a file-descriptor multiplexer

Sep 25, 2022
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file

QueryCSV enable you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to CSV file

Dec 22, 2021
Goful is a CUI file manager written in Go.
Goful is a CUI file manager written in Go.

Goful Goful is a CUI file manager written in Go. Works on cross-platform such as gnome-terminal and cmd.exe. Displays multiple windows and workspaces.

Dec 28, 2022
Read a tar file contents using go1.16 io/fs abstraction
Read a tar file contents using go1.16 io/fs abstraction

go-tarfs Read a tar file contents using go1.16 io/fs abstraction Usage ⚠️ go-tarfs needs go>=1.16 Install: go get github.com/nlepage/go-tarfs Use: pac

Dec 1, 2022
Open Source Continuous File Synchronization
Open Source Continuous File Synchronization

Goals Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals belo

Jan 9, 2023
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

Jan 1, 2023
The best HTTP Static File Server, write with golang+vue
The best HTTP Static File Server, write with golang+vue

gohttpserver Goal: Make the best HTTP File Server. Features: Human-friendly UI, file uploading support, direct QR-code generation for Apple & Android

Dec 30, 2022
Dragonfly is an intelligent P2P based image and file distribution system.
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

Jan 9, 2023
Fast, dependency-free, small Go package to infer the binary file type based on the magic numbers signature

filetype Small and dependency free Go package to infer file and MIME type checking the magic numbers signature. For SVG file type checking, see go-is-

Jan 3, 2023
📂 Web File Browser
📂 Web File Browser

filebrowser provides a file managing interface within a specified directory and it can be used to upload, delete, preview, rename and edit your files.

Jan 9, 2023
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

Jan 2, 2023
File system for GitHub
File system for GitHub

HUBFS · File System for GitHub HUBFS is a read-only file system for GitHub and Git. Git repositories and their contents are represented as regular dir

Dec 28, 2022