Simple in-memory job queue for Golang using worker-based dispatching

GoDoc Build Status Go Report Card codecov

artifex

Simple in-memory job queue for Golang using worker-based dispatching

Documentation here: https://godoc.org/github.com/mborders/artifex

Cron jobs use the robfig/cron library: https://godoc.org/github.com/robfig/cron

Example Usage

// 10 workers, 100 max in job queue
d := artifex.NewDispatcher(10, 100)
d.Start()

d.Dispatch(func() {
  // do something
})

err := d.DispatchIn(func() {
  // do something in 500ms
}, time.Millisecond*500)

// Returns a DispatchTicker
dt, err := d.DispatchEvery(func() {
  // do something every 250ms
}, time.Millisecond*250)

// Stop a given DispatchTicker
dt.Stop()

// Returns a DispatchCron
dc, err := d.DispatchCron(func() {
  // do something every 1s
}, "*/1 * * * * *")

// Stop a given DispatchCron
dc.Stop()

// Stop a dispatcher and all its workers/tickers
d.Stop()
Similar Resources

Go-async - Worker pool (fan-in/fan-out)

Worker pool (fan-in/fan-out) func main() { pool := worker.NewPool(2) ctx := co

Aug 26, 2022

Work pool channlege - An url hash retriever worker pool for getting hash digest for a collection of urls

Code challenge The aim of this project is to provide an url hash retriever worke

Feb 16, 2022

Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines

workerpool Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines. Uses Go 1.18 generics. Notable differe

Oct 5, 2022

Queue is a Golang library for spawning and managing a Goroutine pool

Queue is a Golang library for spawning and managing a Goroutine pool, Alloowing you to create multiple worker according to limit CPU number of machine.

Jan 9, 2023

Queue is a Golang library for spawning and managing a Goroutine pool

Queue is a Golang library for spawning and managing a Goroutine pool, Alloowing you to create multiple worker according to limit CPU number of machine.

Jan 2, 2023

Off heap golang memory pool

Stealthpool stealthpool provides a memory pool that allocates blocks off-heap that will NOT be tracked by the garbage collector. The name stealthpool

Dec 5, 2022

Simply way to control goroutines execution order based on dependencies

Simply way to control goroutines execution order based on dependencies

Goflow Goflow is a simply package to control goroutines execution order based on dependencies. It works similar to async.auto from node.js async packa

Dec 8, 2022

gpool - a generic context-aware resizable goroutines pool to bound concurrency based on semaphore.

gpool - a generic context-aware resizable goroutines pool to bound concurrency. Installation $ go get github.com/sherifabdlnaby/gpool import "github.c

Oct 31, 2022

Golang simple thread pool implementation

Golang Threadpool implementation Scalable threadpool implementation using Go to handle the huge network trafic. Install go get github.com/shettyh/thre

Dec 12, 2022
Comments
  • Module path should be

    Module path should be "github.com/mborders/artifex", not "github.com/borderstech/artifex"

    Background

    Module path is inconsistent with go import path. GO111MODULE=on, as doc said, import "github.com/mborders/artifex", then get this error:

    go: finding module for package github.com/mborders/artifex
    go: downloading github.com/mborders/artifex v0.0.0-20191113201913-8ea03c710bd4
    go: found github.com/mborders/artifex in github.com/mborders/artifex v0.0.0-20191113201913-8ea03c710bd4
    go: test1 imports
            github.com/mborders/artifex: github.com/mborders/[email protected]: parsing go.mod:
            module declares its path as: github.com/borderstech/artifex
                    but was required as: github.com/mborders/artifex 
    

    Solution

    Fix the module path:

    1. Rename the module path to "github.com/mborders/artifex": https://github.com/mborders/artifex/blob/master/go.mod#L1
    module github.com/mborders/artifex
    require (
    	…
    ) 
    
    1. Change the doc document to use import "github.com/borderstech/artifex".
  • Semver tag

    Semver tag

    Hi,

    looks like a very useful module. I tried to pull the latest version but it failed.

    go get -v -u github.com/mborders/[email protected]
    go get github.com/mborders/[email protected]: no matching versions for query "v0.4"
    

    My understanding is that go.mod expects a semver tag and v0.4 doesn't have the patch version. I would assume that tagging v0.4.0 would work without having to change anything.

    For now I can only get v0.3.1 which incidentally follows semver.

  • Throttling

    Throttling

    Awesome library, pretty much exactly what I was looking for! There is one feature I'd still like to have, though.

    I'd like to have the ability to create throttled dispatchers, that is, dispatchers which run no more than X jobs per time (e.g. 5 per minute), regardless of which rate new jobs are created at.

    This could be helpful when sending out large amounts of e-mails, for example. Say, I want to send an e-mail to each of 1,000 users every Friday night, but don't want to send them all at once, but over a period of 30 minutes, to relieve the mail server a bit.

Worker - A Golang library that provides worker pools

Worker A Golang library that provides worker pools. Usage See *_test.go files. T

Apr 15, 2022
goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.

goworker goworker is a Resque-compatible, Go-based background worker. It allows you to push jobs into a queue using an expressive language like Ruby w

Jan 6, 2023
Go simple async worker pool
Go simple async worker pool

??‍?? worker-pool Go simple async worker pool. ?? ABOUT Worker pool is a software design pattern for achieving concurrency of task execution. Maintain

Sep 26, 2022
Deadly simple worker pool

go-worker-pool Deadly simple worker pool Usage package main import ( "errors" workerpool "github.com/zelenin/go-worker-pool" "log" "time" ) func

Dec 10, 2021
golang worker pool , Concurrency limiting goroutine pool

golang worker pool 中文说明 Concurrency limiting goroutine pool. Limits the concurrency of task execution, not the number of tasks queued. Never blocks su

Dec 19, 2022
Golang Implementation of Worker Pool/ Thread Pool

Golang Implementation of Worker Pool/ Thread Pool

Jun 18, 2022
Goworkers - Zero dependency Golang worker pool

Golang Worker Pool Zero dependency golang goroutines pool library. It is useful

Apr 28, 2022
errgroup with goroutine worker limits

neilotoole/errgroup neilotoole/errgroup is a drop-in alternative to Go's wonderful sync/errgroup but limited to N goroutines. This is useful for inter

Dec 15, 2022
Minimalistic and High-performance goroutine worker pool written in Go

pond Minimalistic and High-performance goroutine worker pool written in Go Motivation This library is meant to provide a simple way to limit concurren

Dec 22, 2022
Worker-Pool written in GO

go-workerpool Worker-Pool written in GO Installation go get github.com/agungsid/go-workerpool Usage package main type SampleSeeder struct{} func (s

Jun 10, 2022