86 Resources
Colly Lightning Fast and Elegant Scraping Framework for Gophers Colly provides a clean interface to write any kind of crawler/scraper/spider. With Col
soup Web Scraper in Go, similar to BeautifulSoup soup is a small web scraper package for Go, with its interface highly similar to that of BeautifulSou
Crawlab 中文 | English Installation | Run | Screenshot | Architecture | Integration | Compare | Community & Sponsorship | CHANGELOG | Disclaimer Golang-
Journalist. An RSS aggregator. Download the latest version for macOS, Linux, FreeBSD, NetBSD, OpenBSD & Plan9 here. WARNING: journalist is highly expe
go_spider A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014). QQ群号:337344607 Features Concurrent
crawley Crawls web pages and prints any link it can find. Scan depth (by default - 0) can be configured. features fast SAX-parser (powered by golang.o
Ferret Try it! Docs CLI Test runner Web worker What is it? ferret is a web scraping system. It aims to simplify data extraction from the web for UI te
qq-zone QQ空间爬虫,多协程并发下载相册的相片/视频 前言 QQ相册可以说是存放了好大一部分人生活的点点滴滴,近段时间发现QQ空间莫名会删除短视频或者相片,记得20年的时候也类似新闻报道过,为了快速备份写了此程序,网上看到大部分是使用Python实现的,而且操作过程也都比较繁琐,需要打开网页
goquery - a little like that j-thing, only in Go goquery brings a syntax and a set of features similar to jQuery to the Go language. It is based on Go
GoSpider GoSpider - Fast web spider written in Go Painless integrate Gospider into your recon workflow? Enjoying this tool? Support it's development a
The package includes functions that can scan data from the page into your structs or slice of structs, this allows you to reduce the noise and complexity in your source-code.
Pholcus Pholcus(幽灵蛛)是一款纯 Go 语言编写的支持分布式的高并发爬虫软件,仅用于编程学习与研究。 它支持单机、服务端、客户端三种运行模式,拥有Web、GUI、命令行三种操作界面;规则简单灵活、批量任务并发、输出方式丰富(mysql/mongodb/kafka/csv/excel等
vimego Download Vimeo videos and retrieve metadata. Largely based on yashrathi's vimeo_downloader. Installing go get github.com/raitonoberu/vimego Ple
A powerful browser crawler for web vulnerability scanners
Fetch web pages using headless Chrome, storing all fetched resources including JavaScript files. Run arbitrary JavaScript on many web pages and see the returned values
项目名称 mini-spider 背景 在调研过程中,经常需要对一些网站进行定向抓取。这是一个使用Go语言开发的迷你定向抓取器,实现对种子链接的抓取,并把URL长相符合特定正则表达式的网页保存到磁盘上。 主要功能 实现对种子链接的爬取,并把URL符合特定正则表达式的网页保存到磁盘上。 快速开始 构建
Apollo 💎 A Unix-style personal search engine and web crawler for your digital footprint Demo apollodemo.mp4 Contents Background Thesis Design Archite
Muffet Muffet is a website link checker which scrapes and inspects all pages in a website recursively. Features Massive speed Colored outputs Differen
Jd_Seckill ⚠ 此项目是python jd_seckill 的go版本实现,旨在降低使用门栏和相互学习而创建。 特别声明: 本仓库发布的jd_seckill项目中涉及的任何脚本,仅用于测试和学习研究,禁止用于商业用途,不能保证其合法性,准确性,完整性和有效性,请根据情况自行判断。
predator / 掠食者 基于 fasthttp 开发的高性能爬虫框架 使用 下面是一个示例,基本包含了当前已完成的所有功能,使用方法可以参考注释。 1 创建一个 Crawler import "github.com/go-predator/predator" func main() {
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.
Go-Yahoo-Finance-Daily-Actives - Scrape for the daily actives on yh Finance and save the data to a CSV, and optionally send it to yourself as an email
ErGo Downloader Downloader for saved Reddit images Features Downloads saved images from your Reddit account Optionally unsaves them automatically Opti
自如新房源实时提醒 利用自如网页版查询房源,解析HTML并分析房源,找到新上架房源并推送至钉钉群。 首次初始化加载所选链接的所有房源(不通知) 等待下次任务调度,调度周期时间由taskInterval控制 执行任务,拿到最新房源数据,与上次房源集合进行比对 已存在房源pass,新房源通知钉钉 第一步
cSubsidiary 利用天眼查查询企业子公司 下载地址 介绍 可以通过两种方式查询自己想要的企业子公司 -n 参数:利用给出的关键字先进行模糊查询,然后选出第一个匹配的结果,对该公司进行查询。(方便但是不准确,所以不推荐) -i 参数:利用给出的公司id对该公司进行查询。
Elektra About This Project Elektra is designed to automate the process of invent
About Creeper is a next-generation crawler which fetches web page by creeper script. As a cross-platform embedded crawler, you can use it for your new
skweez skweez (pronounced like "squeeze") spiders web pages and extracts words for wordlist generation. It is basically an attempt to make a more oper
gathertool 轻量级爬虫,接口测试,压力测试框架, 提高开发对应场景的golang程序的效率。
Bathyscaphe dark web crawler Bathyscaphe is a Go written, fast, highly configurable, cloud-native dark web crawler. How to start the crawler To start
Collyzar A distributed redis-based framework for colly. Collyzar provides a very simple configuration and tools to implement distributed crawling/scra
dorkscout dokrscout is a tool to automate the finding of vulnerable applications or secret files around the internet throught google searches, dorksco
Best Room Price Scraper from Booking.com This repo is a tutorial of Large Scale
Go-based search engine URL collector , support Google, Bing, can be based on Google syntax batch collection URL
WhiteChocolateMacademiaNut Description Interacts with Chromium-based browsers' debug port to view open tabs, installed extensions, and cookies. Tested
🔎 scan the internet to find "private" proxies. 🧠 HTTP/SOCKS4/SOCKS5 Proxies. 📌 Installation: - sudo apt-get install git zmap golang
Cirno-go A tool for downloading books from hbooker in Go. Features Login your own account Search books by book name Download books as txt and epub fil
DouYinBot 抖音(中国区)无水印视频、背景音乐、作者ID、作者昵称、作品标题等的全能解析和下载。 写在前面 本项目纯属个人爱好创作 所有视频的版权始终属于「字节跳动」 严禁用于任何商业用途,如果构成侵权概不负责 目前功能 解析无水印视频 解析视频标题 解析作者昵称 解析作者ID 不需要去除多
Snapper A Web microservice for capturing a website's OpenGraph data built in Golang Building Snapper building the binary git clone https://github.com/
crawler-boss 用Go实现抓取Boss直聘职位数据。有几个特点 1.代理防IP被封 2.模拟浏览器,反识别爬虫。 3.控制爬取频率。 4.多协程爬取。 不足之处 1.爬取失败,没有进行重试以及更换IP处理。 2.错误处理 3.代码结构方面进行优化。 交流 && 疑问 如果有任何错误或不懂的
gh-dependents gh command extension to see dependents of your repository. See The GitHub Blog: GitHub CLI 2.0 includes extensions! Install gh extension
Fast golang web crawler for gathering URLs and JavaSript file locations. This is basically a simple implementation of the awesome Gocolly library.
New-World-Auction-House-Crawler Goal of this library is to have a process which grabs New World auction house data in the background while playing the
A crawler/scraper based on golang + colly, configurable via JSON
Super-Simple Scraper This a very thin layer on top of Colly which allows configuration from a JSON file. The output is JSONL which is ready to be impo
A Golang library to scrape lyrics from musixmatch.com (WIP)
Go IMDb Crawler Hit the ⭐ button to show some ❤️ 😃 INSPIRATION 💪 Want to know which celebrities have a common birthday with yours? 👀 Want to get th
🦙 acao acao(阿草), the tool man for data scraping of https://asoul.video/. Deploy to Aliyun serverless function with Raika update_member Update A-SOUL
A quick scraper program that let you export builds on BuildKite as OpenTelemetry data and then send them to honeycomb.io for slice-n-dice high cardinality analysis.
功能 对wireshark生成的抓包文件进行解析,整合TCP数据部分 可用于获取抖音短视频,得到的视频码率为原始码率,并且不带水印 原理 逐一读取.pcap文件的TCP包,按照源目地址进行归类,将服务端返回的数据排序后写入文件 代码内附大量注释,简单易懂。