Weber
Asynchronous fetch unlimited times a website.
Installation
go install github.com/UltiRequiem/weber@latest
Make sure your $PATH
includes $GOPATH/bin
so this command can be used anywhere.
You can also use the binaries from releases.
Usage
Basic Usage:
weber -u http://localhost:3000
This will silently fetch 1 time http://localhost:3000
. You can use --url
instead of -u
.
-l
/ --log
This Option will Log all what is happening, and the parameters that you passed.
weber -u http://localhost:3000 -l
-t
/ --times
weber -u http://localhost:3000 -t 99999 -l
If you want to do too large a number of fetchs, by default the requests will be split into small chunks of 100.
Example: weber -u http://localhost:3000 -t 1000
You told Weber
to make 1000 fetches, but is like that the server is going to ban you if you make all the fetches at the same time. So Weber
divides the 1000 requests in 10 parts of 100 fetches.
You can modify the default value of 100 by using --maxChunkValue
.
-m
/ --maxChunkValue
If you see that with 100 requests per chunk you are not having any problem, you can increase the limit.
weber -u http://localhost:3000 -t 99999 -m 200 -l
License
This project is licensed under the MIT license.