Work pool channlege - An url hash retriever worker pool for getting hash digest for a collection of urls

Code challenge

The aim of this project is to provide an url hash retriever worker pool for getting hash digest for a collection of urls. At the moment, it makes a Get HTTP call for each url, then retrieves MD5 digest of the body of the response.

  • The project is developed in a way that it is flexible for changing the hashing algorithm. The only thing to do is to change the default hashing algorithm using an option.
  • It is flexible to change how we should calculate the digest regarding an url. The only thing to do is to implement httphash.HashRetriever
  • It is possible to specify the number of workers that work in parallel to accomplish the task.

Getting started

In order to use the worker pool follow the below instructions.

1- Install Go programming language (installation manual)

2- Change the working directory to the project directory

3- use following command to build an executable for using the tool

go build -o build/tool cmd/url_hash_retriever/main.go

Then you can use the tool by following command

build/tool 5  http://google.com http://time.ir

The first argument is the number of workers that should accomplish the task, and then you can pass as many urls as you want by other argument.

if you do not provide number of workers its default number is 10.

result

incorrecturl: Get "incorrecturl": unsupported protocol scheme ""
http://google.com: 4232d22639faa16ec9218307919e371b
http://time.ir: 2542908a294bf906f28cc7ee01c24c9e

in the case of occurring an error, instead of hash digest, the error description will be printed after the url.

Structure

The following directory tree is the representation of the structure in which project files and directories are located.

project
│   README.md       project description and guide
│   .gitignore      for specifying files that git must ignore   
│   go.mod          for dependency management
│   go.sum          for dependency management
│      
└─── cmd            Entry point for using the worker pool
│   
└─── tests          unit tests 
│   
└─── tools          packages for having the url hash retriever worker pool

Tests

To run the tests in your development environment that has Go installed, run following command:

go test ./tests/...
Similar Resources

Go-ldap-pool - A simple connection pool for go-ldap

Basic connection pool for go-ldap This little library use the go-ldap library an

Dec 17, 2022

Simple in-memory job queue for Golang using worker-based dispatching

artifex Simple in-memory job queue for Golang using worker-based dispatching Documentation here: https://godoc.org/github.com/mborders/artifex Cron jo

Dec 24, 2022

goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.

goworker goworker is a Resque-compatible, Go-based background worker. It allows you to push jobs into a queue using an expressive language like Ruby w

Jan 6, 2023

errgroup with goroutine worker limits

neilotoole/errgroup neilotoole/errgroup is a drop-in alternative to Go's wonderful sync/errgroup but limited to N goroutines. This is useful for inter

Dec 15, 2022

🐜🐜🐜 ants is a high-performance and low-cost goroutine pool in Go, inspired by fasthttp./ ants 是一个高性能且低损耗的 goroutine 池。

🐜🐜🐜 ants is a high-performance and low-cost goroutine pool in Go, inspired by fasthttp./ ants 是一个高性能且低损耗的 goroutine 池。

A goroutine pool for Go English | 🇨🇳 中文 📖 Introduction Library ants implements a goroutine pool with fixed capacity, managing and recycling a massi

Jan 2, 2023

🐝 A Highly Performant and easy to use goroutine pool for Go

🐝 A Highly Performant and easy to use goroutine pool for Go

gohive Package gohive implements a simple and easy to use goroutine pool for Go Features Pool can be created with a specific size as per the requireme

Sep 26, 2022

gpool - a generic context-aware resizable goroutines pool to bound concurrency based on semaphore.

gpool - a generic context-aware resizable goroutines pool to bound concurrency. Installation $ go get github.com/sherifabdlnaby/gpool import "github.c

Oct 31, 2022

Lightweight Goroutine pool

grpool Lightweight Goroutine pool Clients can submit jobs. Dispatcher takes job, and sends it to first available worker. When worker is done with proc

Dec 6, 2022

Unlimited job queue for go, using a pool of concurrent workers processing the job queue entries

kyoo: A Go library providing an unlimited job queue and concurrent worker pools About kyoo is the phonetic transcription of the word queue. It provide

Dec 21, 2022
Golang Implementation of Worker Pool/ Thread Pool

Golang Implementation of Worker Pool/ Thread Pool

Jun 18, 2022
Worker - A Golang library that provides worker pools

Worker A Golang library that provides worker pools. Usage See *_test.go files. T

Apr 15, 2022
Minimalistic and High-performance goroutine worker pool written in Go

pond Minimalistic and High-performance goroutine worker pool written in Go Motivation This library is meant to provide a simple way to limit concurren

Dec 22, 2022
Go simple async worker pool
Go simple async worker pool

??‍?? worker-pool Go simple async worker pool. ?? ABOUT Worker pool is a software design pattern for achieving concurrency of task execution. Maintain

Sep 26, 2022
Worker-Pool written in GO

go-workerpool Worker-Pool written in GO Installation go get github.com/agungsid/go-workerpool Usage package main type SampleSeeder struct{} func (s

Jun 10, 2022
Deadly simple worker pool

go-worker-pool Deadly simple worker pool Usage package main import ( "errors" workerpool "github.com/zelenin/go-worker-pool" "log" "time" ) func

Dec 10, 2021
Go-async - Worker pool (fan-in/fan-out)

Worker pool (fan-in/fan-out) func main() { pool := worker.NewPool(2) ctx := co

Aug 26, 2022
Goworkers - Zero dependency Golang worker pool

Golang Worker Pool Zero dependency golang goroutines pool library. It is useful

Apr 28, 2022
Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines

workerpool Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines. Uses Go 1.18 generics. Notable differe

Oct 5, 2022
Go-miningcore-pool - Miningcore Pool written in GOlang

Go-Miningcore-Pool (COMING SOON) Miningcore Pool written in GOlang 0x01 Configur

Apr 24, 2022