85 Resources
A powerful browser crawler for web vulnerability scanners
Ferret Try it! Docs CLI Test runner Web worker What is it? ferret is a web scraping system. It aims to simplify data extraction from the web for UI te
soup Web Scraper in Go, similar to BeautifulSoup soup is a small web scraper package for Go, with its interface highly similar to that of BeautifulSou
goquery - a little like that j-thing, only in Go goquery brings a syntax and a set of features similar to jQuery to the Go language. It is based on Go
qq-zone QQ空间爬虫,多协程并发下载相册的相片/视频 前言 QQ相册可以说是存放了好大一部分人生活的点点滴滴,近段时间发现QQ空间莫名会删除短视频或者相片,记得20年的时候也类似新闻报道过,为了快速备份写了此程序,网上看到大部分是使用Python实现的,而且操作过程也都比较繁琐,需要打开网页
Muffet Muffet is a website link checker which scrapes and inspects all pages in a website recursively. Features Massive speed Colored outputs Differen
dorkscout dokrscout is a tool to automate the finding of vulnerable applications or secret files around the internet throught google searches, dorksco
Pholcus Pholcus(幽灵蛛)是一款纯 Go 语言编写的支持分布式的高并发爬虫软件,仅用于编程学习与研究。 它支持单机、服务端、客户端三种运行模式,拥有Web、GUI、命令行三种操作界面;规则简单灵活、批量任务并发、输出方式丰富(mysql/mongodb/kafka/csv/excel等
Journalist. An RSS aggregator. Download the latest version for macOS, Linux, FreeBSD, NetBSD, OpenBSD & Plan9 here. WARNING: journalist is highly expe
Colly Lightning Fast and Elegant Scraping Framework for Gophers Colly provides a clean interface to write any kind of crawler/scraper/spider. With Col
Fetch web pages using headless Chrome, storing all fetched resources including JavaScript files. Run arbitrary JavaScript on many web pages and see the returned values
The package includes functions that can scan data from the page into your structs or slice of structs, this allows you to reduce the noise and complexity in your source-code.
Jd_Seckill ⚠ 此项目是python jd_seckill 的go版本实现,旨在降低使用门栏和相互学习而创建。 特别声明: 本仓库发布的jd_seckill项目中涉及的任何脚本,仅用于测试和学习研究,禁止用于商业用途,不能保证其合法性,准确性,完整性和有效性,请根据情况自行判断。
GoSpider GoSpider - Fast web spider written in Go Painless integrate Gospider into your recon workflow? Enjoying this tool? Support it's development a
Apollo 💎 A Unix-style personal search engine and web crawler for your digital footprint Demo apollodemo.mp4 Contents Background Thesis Design Archite
DouYinBot 抖音(中国区)无水印视频、背景音乐、作者ID、作者昵称、作品标题等的全能解析和下载。 写在前面 本项目纯属个人爱好创作 所有视频的版权始终属于「字节跳动」 严禁用于任何商业用途,如果构成侵权概不负责 目前功能 解析无水印视频 解析视频标题 解析作者昵称 解析作者ID 不需要去除多
A crawler/scraper based on golang + colly, configurable via JSON
Super-Simple Scraper This a very thin layer on top of Colly which allows configuration from a JSON file. The output is JSONL which is ready to be impo
cSubsidiary 利用天眼查查询企业子公司 下载地址 介绍 可以通过两种方式查询自己想要的企业子公司 -n 参数:利用给出的关键字先进行模糊查询,然后选出第一个匹配的结果,对该公司进行查询。(方便但是不准确,所以不推荐) -i 参数:利用给出的公司id对该公司进行查询。
Crawlab 中文 | English Installation | Run | Screenshot | Architecture | Integration | Compare | Community & Sponsorship | CHANGELOG | Disclaimer Golang-
Cirno-go A tool for downloading books from hbooker in Go. Features Login your own account Search books by book name Download books as txt and epub fil
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.
Reactor Crawler Simple CLI content crawler for Joyreactor. He'll find all media content on the page you've provided and save it. If there will be any
Bathyscaphe dark web crawler Bathyscaphe is a Go written, fast, highly configurable, cloud-native dark web crawler. How to start the crawler To start
predator / 掠食者 基于 fasthttp 开发的高性能爬虫框架 使用 下面是一个示例,基本包含了当前已完成的所有功能,使用方法可以参考注释。
predator / 掠食者 基于 fasthttp 开发的高性能爬虫框架 使用 下面是一个示例,基本包含了当前已完成的所有功能,使用方法可以参考注释。 1 创建一个 Crawler import "github.com/go-predator/predator" func main() {
Go-based search engine URL collector , support Google, Bing, can be based on Google syntax batch collection URL
WhiteChocolateMacademiaNut Description Interacts with Chromium-based browsers' debug port to view open tabs, installed extensions, and cookies. Tested
Collyzar A distributed redis-based framework for colly. Collyzar provides a very simple configuration and tools to implement distributed crawling/scra
go_spider A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014). QQ群号:337344607 Features Concurrent
crawley Crawls web pages and prints any link it can find. Scan depth (by default - 0) can be configured. features fast SAX-parser (powered by golang.o
About Creeper is a next-generation crawler which fetches web page by creeper script. As a cross-platform embedded crawler, you can use it for your new
ErGo Downloader Downloader for saved Reddit images Features Downloads saved images from your Reddit account Optionally unsaves them automatically Opti
A quick scraper program that let you export builds on BuildKite as OpenTelemetry data and then send them to honeycomb.io for slice-n-dice high cardinality analysis.
crawler-boss 用Go实现抓取Boss直聘职位数据。有几个特点 1.代理防IP被封 2.模拟浏览器,反识别爬虫。 3.控制爬取频率。 4.多协程爬取。 不足之处 1.爬取失败,没有进行重试以及更换IP处理。 2.错误处理 3.代码结构方面进行优化。 交流 && 疑问 如果有任何错误或不懂的
🦙 acao acao(阿草), the tool man for data scraping of https://asoul.video/. Deploy to Aliyun serverless function with Raika update_member Update A-SOUL
💪 Simple to use. Scrape attendance data with a single command! 🏇 Super fast. A
This is an example project I wrote for a youtube tutorial about webscraping using golang and gocolly It extracts data from a tracking differences webs
gathertool 轻量级爬虫,接口测试,压力测试框架, 提高开发对应场景的golang程序的效率。
skweez skweez (pronounced like "squeeze") spiders web pages and extracts words for wordlist generation. It is basically an attempt to make a more oper
自如新房源实时提醒 利用自如网页版查询房源,解析HTML并分析房源,找到新上架房源并推送至钉钉群。 首次初始化加载所选链接的所有房源(不通知) 等待下次任务调度,调度周期时间由taskInterval控制 执行任务,拿到最新房源数据,与上次房源集合进行比对 已存在房源pass,新房源通知钉钉 第一步
A Golang library to scrape lyrics from musixmatch.com (WIP)
Go IMDb Crawler Hit the ⭐ button to show some ❤️ 😃 INSPIRATION 💪 Want to know which celebrities have a common birthday with yours? 👀 Want to get th
Snapper A Web microservice for capturing a website's OpenGraph data built in Golang Building Snapper building the binary git clone https://github.com/
vimego Download Vimeo videos and retrieve metadata. Largely based on yashrathi's vimeo_downloader. Installing go get github.com/raitonoberu/vimego Ple
Wascra Description Wascra is a tool written in Golang, which lets you extract all relevant Datasheet info from a Warhammer40K (9th edition) faction fr
connpassSearcher connpassSearcher gets ICT event informations Installation When you have installed the Go, Please execute f
Go Site Crawler Go Site Crawler is a simple application written in go that can f
Fast golang web crawler for gathering URLs and JavaSript file locations. This is basically a simple implementation of the awesome Gocolly library.
Elektra About This Project Elektra is designed to automate the process of invent
Best Room Price Scraper from Booking.com This repo is a tutorial of Large Scale
A simple crawler sending Telegram notification when Refurbished Macbook Air / Pro in stock.