47 Resources
Golang scraping-websites Libraries
Tigris is a modern, scalable backend for building real-time websites and apps.
Tigris Data Getting started These instructions will get you through setting up Tigris Data locally as Docker containers. Prerequisites Make sure that
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites by using passive online sources
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites. Designed as a passive framework to be useful for bug bounties and safe for penetration testing.
Newser is a simple utility to generate a pdf with you favorite news articles
Newser A simple utility to crawl some news sites or other resources and download content into a pdf Building Make sure you have config.yaml setup and
Harvit harvests data from different sources (e.g websites, APIs), converts and transforms it.
harvit Harvit harvests data from different sources (e.g websites, APIs), converts and transforms it. Contents harvit Contents Requirements Usage Examp
Moviefetch: a simple program to search and download for movies from websites like 1337x and then stream them
MovieFetch Disclaimer I am NOT responisble for any legal issues or other you enc
Elektra-Auto-Checkout - Utilities to assist in checkout automation of various commercial and retail sites
Elektra About This Project Elektra is designed to automate the process of invent
Monitoriamento-go - Program for monitoring websites in Golang
***Programa para monitoriamento de sites em Go lang. *** No "sitesaqui.txt" colo
Contact-api - API for websites I have designed that have to send unauthenticated email
contact https://aws.amazon.com/premiumsupport/knowledge-center/custom-headers-ap
The world’s fastest framework for building websites.
A Fast and Flexible Static Site Generator built with love by bep, spf13 and friends in Go. Website | Forum | Documentation | Installation Guide | Cont
dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites
Rendora Rendora is a dynamic renderer to provide zero-configuration server-side rendering mainly to web crawlers in order to effortlessly improve SEO
It allows the world to access websites on home computers.
Intranet-Penetration-Go Software function: It allows the world to access websites on home computers. Principle The client runs on a home computer with
Pure Go implementation of fast image search and simple OCR, focused on reading info from screenshots
Lookup It is a nice, simple and fast library which helps you to lookup objects on a screen. It also includes OCR functionality. Using Lookup you can d
Scraping medium blogs to make them loadable with shitty internet and have a pleasant reading experience
Unmedium This project is still WIP We all know medium right? A bunch of JS, wast
Implementing WEB Scraping with Go
WEB Scraping with Go In this project I implement a WEB scraper that create a CSV file with quotes and authors from the Pensador programing Web Page. R
A golang script designed to output the cert information for various websites
gofer gofer is a golang script designed to output the cert information for various websites Example run You can supply multiple sites with port (ie. :
Youtube tutorial about web scraping using golang and Gocolly
This is an example project I wrote for a youtube tutorial about webscraping using golang and gocolly It extracts data from a tracking differences webs
Checking if some websites of interest are online
Web site status checker Checking if some websites of interest are online. A little weekend project. Technologies used Go Gin Current sites supported G
A neat wrapper around the 4chan API for content scraping.
moonarch A neat wrapper around the 4chan API for content scraping. How-To First, get the repository: go get github.com/lazdotdigital/fourscrape. fours
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites by using passive online sources.
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites by using passive online sources. It has a simple modular architecture and is optimized for speed. subfinder is built for doing one thing only - passive subdomain enumeration, and it does that very well.
A boiler-plate like base for people to get started in creating automation software specifically for purchasing items on websites.
Bot-Base Bot-Base is a small project with concepts for most elements of a bot. Feel free to contact me on Twitter with any questions. Contributing Pul
Formrecevr is a simple and lightweight from receiver backend primarily designed for (but not limited to) static websites.
Formrecevr Formrecevr (pronunced "Form receiver") is a simple and lightweight from receiver backend primarily designed for (but not limited to) static
Scrape the web in the eink era. Convert websites into books.
Scrape the web in the eink era. Convert websites into books.
Crawls web pages and prints any link it can find.
crawley Crawls web pages and prints any link it can find. Scan depth (by default - 0) can be configured. features fast SAX-parser (powered by golang.o
Instant online preview of HTML files or websites.
Instant online preview of HTML files or websites.
🦙 acao（阿草）, the tool man for data scraping of https://asoul.video/.
🦙 acao acao（阿草）, the tool man for data scraping of https://asoul.video/. Deploy to Aliyun serverless function with Raika update_member Update A-SOUL
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.
Getting new films from baskino.me
Filmparser Description Every time I want to watch a film, I have to go to baskino.me to check if it is available on torrent trackers to download it th
Aquatone is a tool for visual inspection of websites across a large amount of hosts and is convenient for quickly gaining an overview of HTTP-based attack surface.
Aquatone is a tool for visual inspection of websites across a large amount of hosts and is convenient for quickly gaining an overview of HTTP-based attack surface.
A simple go program which checks if your websites are running and runs forever (stop it with ctrl+c). It takes two optional arguments, comma separated string with urls and an interval.
uptime A simple go program which checks if your websites are running and runs forever (stop it with ctrl+c). It takes two optional arguments: -interva
LinkPage is a FOSS self-hosted alternative to link listing websites such as LinkTree and Campsite.bio
LinkPage LinkPage is a FOSS self-hosted alternative to link listing websites such as LinkTree and Campsite.bio Features Self hostable and open source
Declarative web scraping
Ferret Try it! Docs CLI Test runner Web worker What is it? ferret is a web scraping system. It aims to simplify data extraction from the web for UI te
Elegant Scraper and Crawler Framework for Golang
Colly Lightning Fast and Elegant Scraping Framework for Gophers Colly provides a clean interface to write any kind of crawler/scraper/spider. With Col
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites. Designed as a passive framework to be useful for bug bounties and safe for penetration testing.
Fast passive subdomain enumeration tool. Features • Install • Usage • API Setup • License • Join Discord Subfinder is a subdomain discovery tool that
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Geziyor Geziyor is a blazing fast web crawling and web scraping framework. It can be used to crawl websites and extract structured data from them. Gez
Take screenshots of websites and create PDF from HTML pages using chromium and docker
gochro is a small docker image with chromium installed and a golang based webserver to interact wit it. It can be used to take screenshots of w
Crane - 🐦 A full-text WebAssembley search engine for static websites
Crane 🐦 My blog post: WebAssembly Search Tools for Static Sites Crane is a technical demo is inspired by Stork and uses a near-identical configuratio
A go skeleton for websites.
git-go-websiteskeleton A basic website skeleton in Go that comes with the Gorilla Multiplexer for routing, glog for access and error logging, as well
The world’s fastest framework for building websites.
A Fast and Flexible Static Site Generator built with love by bep, spf13 and friends in Go. Website | Forum | Documentation | Installation Guide | Cont
Extract structured data from web sites. Web sites scraping.
Dataflow kit Dataflow kit ("DFK") is a Web Scraping framework for Gophers. It extracts data from web pages, following the specified CSS Selectors. You
Flexible HTTP command line stress tester for websites and web services
Pewpew Pewpew is a flexible command line HTTP stress tester. Unlike other stress testers, it can hit multiple targets with multiple configurations, si
Extract data or evaluate value from HTML/XML documents using XPath
xquery NOTE: This package is deprecated. Recommends use htmlquery and xmlquery package, get latest version to fixed some issues. Overview Golang packa
⚙️ Convert HTML to Markdown. Even works with entire websites and can be extended through rules.
html-to-markdown Convert HTML into Markdown with Go. It is using an HTML Parser to avoid the use of regexp as much as possible. That should prevent so
A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library
goq Example import ( "log" "net/http" "astuart.co/goq" ) // Structured representation for github file name table type example struct { Title str
Extract structured data from web sites. Web sites scraping.
Dataflow kit Dataflow kit ("DFK") is a Web Scraping framework for Gophers. It extracts data from web pages, following the specified CSS Selectors. You
Elegant Scraper and Crawler Framework for Golang
Colly Lightning Fast and Elegant Scraping Framework for Gophers Colly provides a clean interface to write any kind of crawler/scraper/spider. With Col
A Devtools driver for web automation and scraping
Overview Documentation | API reference Rod is a high-level driver directly based on DevTools Protocol. It's designed for web automation and scraping.
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Antch Antch, inspired by Scrapy. If you're familiar with scrapy, you can quickly get started. Antch is a fast, powerful and extensible web crawling &