Go worker pool

description

This is a generic worker pool for the Go language. It's useful when you want to limit the number of goroutines running in parallel.

installation

go install github.com/stefantalpalaru/pool

compile and run the examples

go run examples/pool_example.go
go run examples/web_crawler.go

The last example is actually an exercise from the Go tour modified to use a worker pool for fetching and processing the URLs. The need to limit the number of concurrent requests in real web scraping scenarios was what prompted the creation of this package.

documentation

You can see the godoc-generated documentation online at godoc.org/github.com/stefantalpalaru/pool but it might be a bit dry.

The best documentation is in the form of an example: pool_example.go.

license

MPL-2.0

credits

Owner
Ștefan Talpalaru
software architect & developer, system administrator
Ștefan Talpalaru
https://github.com/stefantalpalaru/pool
Similar Resources

Deadly simple worker pool

go-worker-pool Deadly simple worker pool Usage package main import ( "errors" workerpool "github.com/zelenin/go-worker-pool" "log" "time" ) func

Dec 10, 2021

Go-async - Worker pool (fan-in/fan-out)

Worker pool (fan-in/fan-out) func main() { pool := worker.NewPool(2) ctx := co

Aug 26, 2022

Goworkers - Zero dependency Golang worker pool

Golang Worker Pool Zero dependency golang goroutines pool library. It is useful

Apr 28, 2022

Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines

workerpool Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines. Uses Go 1.18 generics. Notable differe

Oct 5, 2022

A better Generic Pool (sync.Pool)

A better Generic Pool (sync.Pool)

This package is a thin wrapper over the Pool provided by the sync package. The Pool is an essential package to obtain maximum performance by reducing the number of memory allocations.

Dec 1, 2022

A mining pool proxy tool, support BTC, ETH, ETC, XMR mining pool, etc.

Tier2Pool A mining pool proxy tool, support BTC, ETH, ETC, XMR mining pool, etc. Build I use Ubuntu as a demo. sudo update sudo apt install git make s

Jul 29, 2022

Poweth - Etchash go module intended for use by core-pool (and open-ethereum-pool)

go-etchash Etchash go module intended for use by core-pool (and open-ethereum-po

Jan 20, 2022

Go-miningcore-pool - Miningcore Pool written in GOlang

Go-Miningcore-Pool (COMING SOON) Miningcore Pool written in GOlang 0x01 Configur

Apr 24, 2022

Go-ldap-pool - A simple connection pool for go-ldap

Basic connection pool for go-ldap This little library use the go-ldap library an

Dec 17, 2022

Simple in-memory job queue for Golang using worker-based dispatching

artifex Simple in-memory job queue for Golang using worker-based dispatching Documentation here: https://godoc.org/github.com/mborders/artifex Cron jo

Dec 24, 2022

goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.

goworker goworker is a Resque-compatible, Go-based background worker. It allows you to push jobs into a queue using an expressive language like Ruby w

Jan 6, 2023

errgroup with goroutine worker limits

neilotoole/errgroup neilotoole/errgroup is a drop-in alternative to Go's wonderful sync/errgroup but limited to N goroutines. This is useful for inter

Dec 15, 2022

Job worker service that provides an API to run arbitrary Linux processes.

Job worker service that provides an API to run arbitrary Linux processes.

Job Scheduler Summary Prototype job worker service that provides an API to run arbitrary Linux processes. Overview Library The library (Worker) is a r

May 26, 2022

Worker failover support for PostgreSQL Citus extension using pg_auto_failover.

citus-failover Worker failover support for citus community version using pg_auto_failover. What is this? This is a simple service to monitor changes i

Dec 7, 2022

Kubei is a flexible Kubernetes runtime scanner, scanning images of worker and Kubernetes nodes providing accurate vulnerabilities assessment, for more information checkout:

Kubei is a flexible Kubernetes runtime scanner, scanning images of worker and Kubernetes nodes providing accurate vulnerabilities assessment, for more information checkout:

Kubei is a vulnerabilities scanning and CIS Docker benchmark tool that allows users to get an accurate and immediate risk assessment of their kubernet

Dec 30, 2022

Go worker library

gorkers Examples Quickstart Multiple Go Workers Passing Fields Getting Started Import import ( "github.com/guilhem/gorkers" ) Create a worker func

Sep 30, 2022

Go lang concurrency with channels and worker pools using sync.Pools.

Go Concurrency Go lang concurrency with channels and worker pools using sync.Pools. Concurrency Concurrency is an ability of a program to do multiple

Oct 31, 2021

Support variable parameters task send worker

go-worker-pool Support variable parameters task send worker 调用方式 go get github.com/214200196/gwp/v1 import ( "fmt" "github.com/214200196/gwp/v1" "

Dec 16, 2021

Snowflake algorithm generation worker Id sequence

sequence snowflake algorithm generation worker Id sequence 使用雪花算法生成ID，生成100万个只需要

Jan 21, 2022
Comments
  • Case for non-determistic number of jobs

    Case for non-determistic number of jobs

    hi - It doesn't look this project is not in active development, but I will ask anyway. I have a use case where the number of jobs is non-deterministic: in the sense that each Job may or may not generate new Jobs. The termination condition is no new jobs are generated by all submitted jobs. Can this pool structure support such use case? TIA.

  • Any way to get the id of the worker

    Any way to get the id of the worker

    Hi Firstly, this is a nice utility. Thanx Qs: Im creating 5 workers using pool.New. How do I pass in the id lets say ids are 1 to 5. Or how do I get the worker ids in the Job result?

  • use atomic instead of mutex

    use atomic instead of mutex

    Its a good idea that rename Next_job_id to next_job_id, public is not thread safe. But i still think atomic is better, simplify and effective, is this proposal acceptable to you?

Related tags
Miscellaneous pool
Algorand offchain worker with golang

Algorand offchain worker with golang

Sep 12, 2022
a generic object pool for golang

Go Commons Pool The Go Commons Pool is a generic object pool for Golang, direct rewrite from Apache Commons Pool. Features Support custom PooledObject

Jan 5, 2023
Gowl is a process management and process monitoring tool at once. An infinite worker pool gives you the ability to control the pool and processes and monitor their status.
 Gowl is a process management and process monitoring tool at once. An infinite worker pool gives you the ability to control the pool and processes and monitor their status.

Gowl is a process management and process monitoring tool at once. An infinite worker pool gives you the ability to control the pool and processes and monitor their status.

Nov 10, 2022
Golang Implementation of Worker Pool/ Thread Pool

Golang Implementation of Worker Pool/ Thread Pool

Jun 18, 2022
Work pool channlege - An url hash retriever worker pool for getting hash digest for a collection of urls

Code challenge The aim of this project is to provide an url hash retriever worke

Feb 16, 2022
Worker - A Golang library that provides worker pools

Worker A Golang library that provides worker pools. Usage See *_test.go files. T

Apr 15, 2022
Minimalistic and High-performance goroutine worker pool written in Go

pond Minimalistic and High-performance goroutine worker pool written in Go Motivation This library is meant to provide a simple way to limit concurren

Dec 22, 2022
Go simple async worker pool
 Go simple async worker pool

??‍?? worker-pool Go simple async worker pool. ?? ABOUT Worker pool is a software design pattern for achieving concurrency of task execution. Maintain

Sep 26, 2022
Go worker pool

description This is a generic worker pool for the Go language. It's useful when you want to limit the number of goroutines running in parallel. instal

Jul 4, 2021
Worker-Pool written in GO

go-workerpool Worker-Pool written in GO Installation go get github.com/agungsid/go-workerpool Usage package main type SampleSeeder struct{} func (s

Jun 10, 2022