Download an entire website with Go

Go Download Web

A simple command-line application to download an entire online website, including CSS, JSS, and other assets. Coded with Go.

Project status

There are still some to-do's, and some refactoring is needed, but the app is already functional.

Installation

There is nothing special to do here, just download the code, build it as you would do with any other Go app, and you are set to go.

$ git clone github.com/antsanchez/go-download-web
$ cd go-download-web
$ go build

Usage

# Default mode:
$ ./go-download-web -u https://example.com

# Setting the number of concurrent connections to 10:
$ ./go-download-web -u https://example.com -s 10

# Also scrapping URLs with query params:
$ ./go-download-web -u https://example.com -q

# Change the domain name of the downloaded copy:
$ ./go-download-web -u https://example.com -new https://newname.com

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

Author

Antonio Sánchez

License

MIT

Owner
Antonio Sánchez
Just another web developer
Antonio Sánchez
Similar Resources

Monitor your Website and APIs from your Computer. Get Notified through Slack, E-mail when your server is down or response time is more than expected.

Monitor your Website and APIs from your Computer. Get Notified through Slack, E-mail when your server is down or response time is more than expected.

StatusOK Monitor your Website and APIs from your computer.Get notified through Slack or E-mail when your server is down or response time is more than

Dec 27, 2022

:construction: Closed. A website and user system (Gin/Backbone).

A website and user system starter. Implemented with gin and Backbone. Gowall is port of Drywall Go Node.js Repository here Drywall Site Gowall Drywall

Jul 6, 2020

This service provides authenticated access to a static website hosted in an s3 bucket.

This service provides authenticated access to a static website hosted in an s3 bucket.

website-openid-proxy This service provides OpenID authenticated access to a static website hosted in an s3 bucket. It is designed to be a simple way t

Jan 19, 2022

Standup a new website in 2 commands.

Standup a new website in 2 commands.

Page CLI Getting Started Download latest release for your operating system. Install by unzipping it and moving the file to a directory included in you

Nov 18, 2022

Backpulse's core. Backpulse is an API Based CMS. Build you own website without worrying about the content administration system.

Backpulse core Backpulse is an API Based / Headless CMS. Your site's content is accessible directly via our RESTful API, on any web framework and any

Sep 11, 2022

Fast website link checker in Go

Fast website link checker in Go

Muffet Muffet is a website link checker which scrapes and inspects all pages in a website recursively. Features Massive speed Colored outputs Differen

Dec 26, 2022

导航网站生成器(Navigation website generator)

导航网站生成器(Navigation website generator)

gena 导航网站生成器 | English Document 安装 一键生成(推荐) 从 gena-template 自动生成并自动部署到 GitHub Pages 源码安装 go1.16 required go get -u github.com/x1ah/gena/cmd/gena gen

Nov 20, 2022

Collection of tools to interact with Intigriti website

Collection of tools to interact with Intigriti website

Small tool, written in Go, that constantly monitors Activity feed on https://app.intigriti.com Dashboard page and sends Slack/Discord notifications on

Dec 23, 2022

An autoclaimer for the website We Heart It

weheartit-claimer An autoclaimer for the website We Heart It Table of contents Setting up the claimer Understanding the configuration files Setup Firs

May 26, 2021

⚗ The most advanced CLI template on earth! Featuring automatic releases, website generation and a custom CI-System out of the box.

⚗ The most advanced CLI template on earth! Featuring automatic releases, website generation and a custom CI-System out of the box.

cli-template ✨ ⚗ A template for beautiful, modern, cross-platform compatible CLI tools written with Go! Getting Started | Wiki This template features

Dec 4, 2022

Self-hosted video-hosting website and video archival manager for Niconico, Bilibili, and Youtube

Self-hosted video-hosting website and video archival manager for Niconico, Bilibili, and Youtube

Self-hosted video-hosting website and video archival manager for Niconico, Bilibili, and Youtube

Jan 1, 2023

CDN for the Tech With Tim website using Go

CDN for the Tech With Tim website using Go

Tech With Tim - CDN CDN for the Tech With Tim website using Go 📝 Table of Contents 🏁 Getting Started Environment variables Running 🐳 Running with D

Apr 29, 2022

Access my website from the terminal with SSH!

Access my website from the terminal with SSH!

daniel.is-a.dev (ssh version) What is this? I built a SSH server written in Golang that lets you basically view my website all from the terminal. You

Nov 11, 2022

Asynchronous fetch unlimited times a website.

Asynchronous fetch unlimited times a website.

Apr 7, 2022

Simple website fingerprinting tool

onefinger 一个简单的指纹识别工具,规则提取自goby并稍作修改,某个知识星球的作业 Usage Usage: onefinger [-v] | (-t=target... | --tf=targetFile) [--threads=threads] [--timeout=ti

Mar 16, 2022

🪐 ✨ GO server for lorem.space website ✨ 🪐

Lorem.space server The backend of https://lorem.space lives here. Features Scan the directory & categorize the image resources (only JPEG for now) Pro

Oct 6, 2022

🔥 Hugo website builder, Hugo themes & Hugo CMS. No code, build with widgets!

🔥 Hugo website builder, Hugo themes & Hugo CMS. No code, build with widgets!

Wowchemy: the website builder for Hugo Join 750,000+ Sites. No Code. Easily Create Future-Proof Websites ✏️ 📰 🚀 🔥 1. Create any kind of website

Jan 9, 2023

A tool to solve DNS pollution of GitHub website. Query the real IP address of domain names such as github.com, and refresh the domain name setting of the system hosts file.

githubDNS Target A tool to solve DNS pollution of GitHub website. Query the real IP address of domain names such as github.com, and refresh the domain

Oct 14, 2021

Opinionated boilerplate Golang HTTP server with CORS, OPA, Prometheus, rate-limiter for API and static website.

Opinionated boilerplate Golang HTTP server with CORS, OPA, Prometheus, rate-limiter for API and static website.

Teal.Finance/Server Opinionated boilerplate HTTP server with CORS, OPA, Prometheus, rate-limiter… for API and static website. Origin This library was

Nov 3, 2022
Comments
  • URL Update

    URL Update

    Change the Installation URL from: $ git clone github.com/antsanchez/go-download-web to: $ git clone https://github.com/antsanchez/go-download-web

    git clone will not accept without protocol portion

🥒 Simple REST-API product service (download products in .csv file)

?? Simple REST-API product service (download products in .csv file)

Nov 28, 2021
zipspy - a CLI tool to extract files from zip archives in S3 without needing to download the entire archive

Zipspy allows you interact with ZIP archives stored in remote locations without requiring a local copy. For example, you can list the filenames in an S3 ZIP archive, download a subset of files, search and retrieve files with regular expressions, and more!

Feb 19, 2022
Scraper to download school attendance data from the DfE's statistics website
Scraper to download school attendance data from the DfE's statistics website

?? Simple to use. Scrape attendance data with a single command! ?? Super fast. A

Mar 31, 2022
Quickly clone an entire org/users repositories into one directory - Supports GitHub, GitLab, Bitbucket, and more
Quickly clone an entire org/users repositories into one directory - Supports GitHub, GitLab, Bitbucket, and more

ghorg ghorg allows you to quickly clone all of an orgs, or users repos into a single directory. This can be useful in many situations including Search

Jan 1, 2023
⚙️ Convert HTML to Markdown. Even works with entire websites and can be extended through rules.
⚙️ Convert HTML to Markdown. Even works with entire websites and can be extended through rules.

html-to-markdown Convert HTML into Markdown with Go. It is using an HTML Parser to avoid the use of regexp as much as possible. That should prevent so

Jan 6, 2023
A command-line to create a pull request to review the entire content of a Github repository.

Pull Request Me Pull Request Me (PRMe) creates a pull request for the entire content of a Github repository. This is useful to solicit review comments

Nov 2, 2021
erchive is a go program that compresses and encrypts files and entire directories into .zep files (encrypted zip files).

erchive/zep erchive is a go program that compresses and encrypts files and entire directories into .zep files (encrypted zip files). it compresses usi

May 16, 2022
Mattermost is an open source platform for secure collaboration across the entire software development lifecycle.
Mattermost is an open source platform for secure collaboration across the entire software development lifecycle.

Mattermost is an open source platform for secure collaboration across the entire software development lifecycle. This repo is the primary source for c

Jan 2, 2023
DeepCopy a portable app that allows you to copy all forms of specified file types from your entire file system of the computer

DeepCopy a portable app that allows you to copy all forms of specified file types from your entire file system of the computer

Dec 20, 2021
Generate vector tiles for the entire planet on relatively low spec hardware.
Generate vector tiles for the entire planet on relatively low spec hardware.

Sequentially Generate Planet Mbtiles Sequentially generate and merge an entire planet.mbtiles vector tileset on low memory/power devices for free. com

Dec 21, 2022