Asynchronous fetch unlimited times a website.

Weber

GitMoji License: MIT Lines Of Code CodeQL Build Go Report Card wakatime

Asynchronous fetch unlimited times a website.

Installation

go install github.com/UltiRequiem/weber@latest

Make sure your $PATH includes $GOPATH/bin so this command can be used anywhere.

You can also use the binaries from releases.

Usage

Basic Usage:

weber -u http://localhost:3000

This will silently fetch 1 time http://localhost:3000. You can use --url instead of -u.

-l / --log

This Option will Log all what is happening, and the parameters that you passed.

weber -u http://localhost:3000 -l

-t / --times

weber -u http://localhost:3000 -t 99999 -l

If you want to do too large a number of fetchs, by default the requests will be split into small chunks of 100.

Example: weber -u http://localhost:3000 -t 1000

You told Weber to make 1000 fetches, but is like that the server is going to ban you if you make all the fetches at the same time. So Weber divides the 1000 requests in 10 parts of 100 fetches.

You can modify the default value of 100 by using --maxChunkValue.

-m / --maxChunkValue

If you see that with 100 requests per chunk you are not having any problem, you can increase the limit.

weber -u http://localhost:3000 -t 99999 -m 200 -l

License

This project is licensed under the MIT license.

Owner
Eliaz Bobadilla
14 yo • Focuses on Golang & JavaScript. Makes Web apps, CLI tools, node & deno packages.
Eliaz Bobadilla
Similar Resources

Flowgraph package for scalable asynchronous system development

flowgraph Getting Started go get -u github.com/vectaport/flowgraph go test Links Wiki Slides from Minneapolis Golang Meetup, May 22nd 2019 Overview F

Dec 22, 2022

Library for enabling asynchronous health checks in your service

go-health A library that enables async dependency health checking for services running on an orchestrated container platform such as kubernetes or mes

Jan 4, 2023

Go asynchronous simple function utilities, for managing execution of closures and callbacks

Go asynchronous simple function utilities, for managing execution of closures and callbacks

⚙️ gollback gollback - Go asynchronous simple function utilities, for managing execution of closures and callbacks 📖 ABOUT Contributors: Rafał Lorenz

Dec 29, 2022

goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.

goworker goworker is a Resque-compatible, Go-based background worker. It allows you to push jobs into a queue using an expressive language like Ruby w

Jan 6, 2023

Hunch provides functions like: All, First, Retry, Waterfall etc., that makes asynchronous flow control more intuitive.

Hunch provides functions like: All, First, Retry, Waterfall etc., that makes asynchronous flow control more intuitive.

Hunch Hunch provides functions like: All, First, Retry, Waterfall etc., that makes asynchronous flow control more intuitive. About Hunch Go have sever

Dec 8, 2022

A Microservice Toolkit from The New York Times

A Microservice Toolkit from The New York Times

Gizmo Microservice Toolkit This toolkit provides packages to put together server and pubsub daemons with the following features: Standardized configur

Dec 27, 2022

A Go (golang) package providing high-performance asynchronous logging, message filtering by severity and category, and multiple message targets.

ozzo-log Other languages 简体中文 Русский Description ozzo-log is a Go package providing enhanced logging support for Go programs. It has the following fe

Dec 17, 2022

Seelog is a native Go logging library that provides flexible asynchronous dispatching, filtering, and formatting.

Seelog Seelog is a powerful and easy-to-learn logging framework that provides functionality for flexible dispatching, filtering, and formatting log me

Jan 3, 2023

Machinery is an asynchronous task queue/job queue based on distributed message passing.

Machinery is an asynchronous task queue/job queue based on distributed message passing.

Machinery Machinery is an asynchronous task queue/job queue based on distributed message passing. V2 Experiment First Steps Configuration Lock Broker

Jan 7, 2023

Gin is a HTTP web framework written in Go (Golang). It features a Martini-like API with much better performance -- up to 40 times faster. If you need smashing performance, get yourself some Gin.

Gin is a HTTP web framework written in Go (Golang). It features a Martini-like API with much better performance -- up to 40 times faster. If you need smashing performance, get yourself some Gin.

Gin Web Framework Gin is a web framework written in Go (Golang). It features a martini-like API with performance that is up to 40 times faster thanks

Jan 2, 2023

A Microservice Toolkit from The New York Times

A Microservice Toolkit from The New York Times

Gizmo Microservice Toolkit This toolkit provides packages to put together server and pubsub daemons with the following features: Standardized configur

Jan 7, 2023

A fast and easy to use URL health checker ⛑️ Keep your links healthy during tough times

A fast and easy to use URL health checker ⛑️ Keep your links healthy during tough times

AreYouOK? A minimal, fast & easy to use URL health checker Who is AreYouOk made for ? OSS Package Maintainers 📦️

Oct 7, 2022

Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and deadline support. MIT Licensed.

socket Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and d

Dec 14, 2022

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

Cadence Visit cadenceworkflow.io to learn about Cadence. This repo contains the source code of the Cadence server. To implement workflows, activities

Jan 9, 2023

Fully asynchronous, structured, pluggable logging for Go.

logr Logr is a fully asynchronous, contextual logger for Go. It is very much inspired by Logrus but addresses two issues: Logr is fully asynchronous,

Dec 28, 2022

Machinery is an asynchronous task queue/job queue based on distributed message passing.

Machinery is an asynchronous task queue/job queue based on distributed message passing.

Machinery Machinery is an asynchronous task queue/job queue based on distributed message passing. V2 Experiment First Steps Configuration Lock Broker

Dec 24, 2022

Easily schedule commands to run multiple times at set intervals (like a cronjob, but with one command)

hakcron Easily schedule commands to run multiple times at set intervals (like a cronjob, but for a single command) Description hakcron allows you to r

Aug 17, 2022

A Microservice Toolkit from The New York Times

A Microservice Toolkit from The New York Times

Gizmo Microservice Toolkit This toolkit provides packages to put together server and pubsub daemons with the following features: Standardized configur

May 31, 2021

GTA(Go Task Async) is a lightweight reliable asynchronous task and transaction message library for Golang

GTA (Go Task Async) is a lightweight and reliable asynchronous task and transaction message library for by golang.

Jun 4, 2022
timea.go (did you see what I did there?) is a simple library to print given times in

timea.go timea.go (did you see what I did there?) is a simple library to print given times in "time ago" manner. Usage Get it: go get github.com/caarl

Sep 29, 2022
Go-fetch-words - Fetch 5 letter words from dictionary.com

Go-fetch-words This GO app fetches 5 letter words from dictionary.com and saves

Oct 16, 2022
Fetch-npm-package - A small utility that can be used to fetch a given version of a NPM package

Use fetch-npm-package <package> <version> <output-dir> E.g. fetch-npm-package is

May 21, 2022
Unlimited job queue for go, using a pool of concurrent workers processing the job queue entries

kyoo: A Go library providing an unlimited job queue and concurrent worker pools About kyoo is the phonetic transcription of the word queue. It provide

Dec 21, 2022
:speedboat: a limited consumer goroutine or unlimited goroutine pool for easier goroutine handling and cancellation

Package pool Package pool implements a limited consumer goroutine or unlimited goroutine pool for easier goroutine handling and cancellation. Features

Jan 1, 2023
An encrypted object storage system with unlimited space backed by Telegram.

TGStore An encrypted object storage system with unlimited space backed by Telegram. Please only upload what you really need to upload, don't abuse any

Nov 28, 2022
Abusing Discord for unlimited cloud storage

Discord Cloud Storage Abusing Discord's servers for unlimited cloud storage! So, what is this? Infamous 8MB limit for non-nitro users can get pretty a

Nov 26, 2022
A set of components that can be composed into a highly available metric system with unlimited storage capacity
A set of components that can be composed into a highly available metric system with unlimited storage capacity

Overview Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added

Oct 20, 2021
Just Dance Unlimited mock-up server written on Golang and uses a popular Gin framework for Go.

BDCS Just Dance Unlimited mock-up server written on Golang and uses a popular Gin framework for Go. Features Security Authorization works using UbiSer

Nov 10, 2021
Go package for calculating the sunrise and sunset times for a given location

go-sunrise Go package for calculating the sunrise and sunset times for a given location based on this method. Usage To calculate sunrise and sunset ti

Dec 13, 2022