go task pool

Task Pool

Task Pool 是一个易于使用且高度可配置的 golang类库,专门用于任务的管理&执行,支持自定义次数的重发。

功能特点

  • 线程安全 - task pool 内所有的方法以及暴露的接口都是线程安全的
  • 异步发送 - 调用 PushTask 方法后回立即返回,任务将会被传递到io线程中异步发送,不阻塞用户操作。
  • 失败重试 - 用户可以通过设置初始化的参数来指定任务执行失败的次数,超过重试次数将被投递到失败队列。
  • 优雅关闭 - 用户调用关闭方法进行关闭时,task pool 会将所有其缓存的数据进行发送,防止任务丢失。
  • 结果可控 - 用户可以自己实现 Task 接口,里面包含任务执行成功和失败后调用的方法,以及任务具体执行的操作。

安装

go get github.com/overtalk/task

使用步骤

1.配置Config

  • Config 是提供给用户的配置类,用于配制任务执行策略,您可以根据不同的需求设置不同的值,具体的参数含义如文章尾的配置详解所示。

启动 taskPool 进程

c := task.GetDefaultConfig()
taskPool, err := task.NewTaskPool(c)
if err != nil {
    log.Fatal(err)
}
taskPool.Start() // 启动 task pool 实例

3.调用 PushTask 方法加入任务

  • 任务需要自定义,实现一下接口即可
type Task interface {
	Execute() error
	CallBack(result *Result)
}
  • task pool 中提供了 PushTask 方法供用户将任务加入到任务池中
// 执行任务
for i := 0; i < 10; i++ {
    t := &myTask{
        name:  fmt.Sprintf("task - %d", i),
        times: 0,
    }
    taskPool.PushTask(t)
}

4.关闭 taskPool

  • taskPool 提供安全关闭的方式,会等待 taskPool 中缓存的所有的任务全部发送完成以后在关闭 taskPool
taskPool.SafeClose()

5.获取任务执行结果

  • taskPool 中的任务执行是异步的,所以需要用户实现 Task 接口,去获得每次执行的结果。

  • 实现 Task 接口需要实现其中的Execute()方法和CallBack()方法,两个方法分别会在任务执行成功或失败的时候去调用,两个方法会都会接收一个Result 实例,用户可以根据Result实例在CallBack回调方法中去获得每次任务执行的结果。下面写了一个简单的使用样例。

type myTask struct {
	name  string
	times int
}

func (myTask *myTask) Execute() error {
	fmt.Printf("*Execute* [%s], times = [%d]\n", myTask.name, myTask.times)

	if myTask.times == 0 {
		myTask.times++
		return errors.New("fail")
	}

	return nil
}

func (myTask *myTask) CallBack(result *task.Result) {
    if result.IsSuccessful() {
        fmt.Printf("*success* [%s]! times = [%d]\n", myTask.name, myTask.times)
    } else {
        fmt.Printf("*fail* [%s]! times = [%d]\n", myTask.name, myTask.times)
    }
}
  • 用户可以根据自己的需求调用Result实例提供的方法来获取任务执行结果信息。

taskPool 配置详解

参数 类型 描述
MaxTaskNum Int 单个 taskPool 实例能缓存的任务数量上限,默认为 1000。
MaxBlockSec Int 如果 taskPool 任务数量已达数量上限,调用者在 PushTask 方法上的最大阻塞时间,默认为 60 秒。
如果超过这个时间后任务数量没有空余,PushTask 方法会抛出TimeoutException。如果将该值设为0,当任务数量无法得到满足时,PushTask 方法会立即抛出 TimeoutException。如果您希望 PushTask 方法一直阻塞直到任务数量得到满足,可将该值设为负数。
MaxIoWorkerNum Int64 单个 taskPool 能并发的最多groutine的数量,默认为50,该参数用户可以根据自己实际服务器的性能去配置。
MaxRetryTimes Int 如果某个 task 首次执行失败,能够对其重试的次数,默认为 10 次。
如果 retries 小于等于 0,该 ProducerBatch 首次发送失败后将直接进入失败队列。
BaseRetryBackOffMs Int64 首次重试的退避时间,默认为 100 毫秒。 taskPool 采样指数退避算法,第 N 次重试的计划等待时间为 baseRetryBackOffMs * 2^(N-1)。
MaxRetryBackOffMs Int64 重试的最大退避时间,默认为 50 秒。

问题反馈

如果您在使用过程中遇到了问题,可以创建 GitHub Issue

Owner
qinhan
Golang Rookie.
qinhan
Similar Resources

Go distributed task scheduler

Go distributed task scheduler

Nov 13, 2021

A cross-platform task runner for executing commands and generating files from templates

A cross-platform task runner for executing commands and generating files from templates

Orbit A cross-platform task runner for executing commands and generating files from templates Orbit started with the need to find a cross-platform alt

Oct 22, 2022

Tasks - Golang CLI, Task manager

Tasks Golang CLI, Task manager Prerequisites Golang Setup environment variables

Jan 30, 2022

A better Generic Pool (sync.Pool)

A better Generic Pool (sync.Pool)

This package is a thin wrapper over the Pool provided by the sync package. The Pool is an essential package to obtain maximum performance by reducing the number of memory allocations.

Dec 1, 2022

Gowl is a process management and process monitoring tool at once. An infinite worker pool gives you the ability to control the pool and processes and monitor their status.

Gowl is a process management and process monitoring tool at once. An infinite worker pool gives you the ability to control the pool and processes and monitor their status.

Gowl is a process management and process monitoring tool at once. An infinite worker pool gives you the ability to control the pool and processes and monitor their status.

Nov 10, 2022

A mining pool proxy tool, support BTC, ETH, ETC, XMR mining pool, etc.

Tier2Pool A mining pool proxy tool, support BTC, ETH, ETC, XMR mining pool, etc. Build I use Ubuntu as a demo. sudo update sudo apt install git make s

Jul 29, 2022

Poweth - Etchash go module intended for use by core-pool (and open-ethereum-pool)

go-etchash Etchash go module intended for use by core-pool (and open-ethereum-po

Jan 20, 2022

Golang Implementation of Worker Pool/ Thread Pool

Golang Implementation of Worker Pool/ Thread Pool

Jun 18, 2022

Go-miningcore-pool - Miningcore Pool written in GOlang

Go-Miningcore-Pool (COMING SOON) Miningcore Pool written in GOlang 0x01 Configur

Apr 24, 2022

Go-ldap-pool - A simple connection pool for go-ldap

Basic connection pool for go-ldap This little library use the go-ldap library an

Dec 17, 2022

Work pool channlege - An url hash retriever worker pool for getting hash digest for a collection of urls

Code challenge The aim of this project is to provide an url hash retriever worke

Feb 16, 2022

go task pool

Task Pool Task Pool 是一个易于使用且高度可配置的 golang类库,专门用于任务的管理&执行,支持自定义次数的重发。 功能特点 线程安全 - task pool 内所有的方法以及暴露的接口都是线程安全的 异步发送 - 调用 PushTask 方法后回立即返回,任务将会被传递到io

Dec 29, 2022

Routines was a fixed number thread pool to process the user task, and it would respawn a corresponding new thread when panic

Routines Routines was a fixed number thread pool to process the user task, and it would respawn a corresponding new thread when panic. It supports the

Dec 16, 2021

Task Timer (tt) is a dead simple TUI task timer

Task Timer (tt) is a dead simple TUI task timer

tasktimer Task Timer (tt) is a dead simple TUI task timer Usage To get started, just run tt: tt You'll be presented with something like this: You can

Dec 21, 2022

GTA(Go Task Async) is a lightweight reliable asynchronous task and transaction message library for Golang

GTA (Go Task Async) is a lightweight and reliable asynchronous task and transaction message library for by golang.

Jun 4, 2022

tasq is a simple HTTP-based task queue. Each task is represented as a string

tasq tasq is a simple HTTP-based task queue. Each task is represented as a string (it could be anything). Tasks are pushed to the queue via an HTTP en

Nov 3, 2022

Gotask - A simple task queue is stripped when the program is written to achieve the task delivery function

Gotask - A simple task queue is stripped when the program is written to achieve the task delivery function

gotask The simple task queue is stripped when the program is written to achieve

Feb 14, 2022

🐜🐜🐜 ants is a high-performance and low-cost goroutine pool in Go, inspired by fasthttp./ ants 是一个高性能且低损耗的 goroutine 池。

🐜🐜🐜 ants is a high-performance and low-cost goroutine pool in Go, inspired by fasthttp./ ants 是一个高性能且低损耗的 goroutine 池。

A goroutine pool for Go English | 🇨🇳 中文 📖 Introduction Library ants implements a goroutine pool with fixed capacity, managing and recycling a massi

Jan 2, 2023

🐝 A Highly Performant and easy to use goroutine pool for Go

🐝 A Highly Performant and easy to use goroutine pool for Go

gohive Package gohive implements a simple and easy to use goroutine pool for Go Features Pool can be created with a specific size as per the requireme

Sep 26, 2022
Related tags
Gotask - A simple task queue is stripped when the program is written to achieve the task delivery function
Gotask - A simple task queue is stripped when the program is written to achieve the task delivery function

gotask The simple task queue is stripped when the program is written to achieve

Feb 14, 2022
Distributed Task Scheduling System|分布式定时任务调度平台
Distributed Task Scheduling System|分布式定时任务调度平台

Crocodile Distributed Task Scheduling System English | 中文 Introduction A distributed task scheduling system based on Golang that supports http request

Jan 5, 2023
Go based task runner

Grift Grift is a very simple library that allows you to write simple "task" scripts in Go and run them by name without having to write big main type o

Nov 21, 2022
Celery Distributed Task Queue in Go
Celery Distributed Task Queue in Go

gocelery Go Client/Server for Celery Distributed Task Queue Why? Having been involved in several projects migrating servers from Python to Go, I have

Jan 1, 2023
Machinery is an asynchronous task queue/job queue based on distributed message passing.
Machinery is an asynchronous task queue/job queue based on distributed message passing.

Machinery Machinery is an asynchronous task queue/job queue based on distributed message passing. V2 Experiment First Steps Configuration Lock Broker

Dec 24, 2022
high performance distributed task scheduling system, Support multi protocol scheduling tasks
 high performance distributed task scheduling system, Support multi protocol scheduling tasks

high performance distributed task scheduling system, Support multi protocol scheduling tasks

Dec 2, 2022
Chrono is a scheduler library that lets you run your task and code periodically
Chrono is a scheduler library that lets you run your task and code periodically

Chrono is a scheduler library that lets you run your tasks and code periodically. It provides different scheduling functionalities to make it easier t

Dec 26, 2022
YTask is an asynchronous task queue for handling distributed jobs in golang
YTask is an asynchronous task queue for handling distributed jobs in golang

YTask is an asynchronous task queue for handling distributed jobs in golang

Dec 24, 2022
goInterLock is golang job/task scheduler with distributed locking mechanism (by Using Redis🔒).
goInterLock is golang job/task scheduler with distributed locking mechanism (by Using Redis🔒).

goInterLock is golang job/task scheduler with distributed locking mechanism. In distributed system locking is preventing task been executed in every instant that has the scheduler,

Dec 5, 2022
go-sche is a golang library that lets you schedule your task to be executed later.

go-sche is a golang library that lets you schedule your task to be executed later.

Dec 24, 2022