Scheduler: Go jobs execution system

Scheduler

Go jobs execution system. Inspired by CI/CD and Unity task scheduler.
Created to be used for kyoto lifecycle management, but also can be easily used for other purposes.

Features

  • Define job execution groups
  • Configure jobs to be dependent on another job or entire group
  • Configure workers count
  • Simple shortcuts for parallel processing (for, foreach)

Reasons?

Managing jobs with plain Go code is not easy. You need to write a lot of boilerplate code, keep track of dependencies between jobs, manually manage job execution order, and so on. This library is designed to hide all of these complexities under the hood.

Examples

Quick Start

...

func main() {
    s := scheduler.New()
    s.Workers = 2

    // This jobs will be executed in parallel
    s.Add(scheduler.Job{
        Func: func() error {
            log.Println("I'm a job 1")
        },
    })
    s.Add(scheduler.Job{
        Func: func() error {
            log.Println("I'm a job 2")
        },
    })

    s.Execute()
}

Dependencies

...

func main() {
    s := scheduler.New()
    s.Workers = 2

    // This jobs will be executed in parallel
    s.Add(scheduler.Job{
        Group: "group0",
        Func: func() error {
            log.Println("I'm a job 1")
        },
    })
    s.Add(scheduler.Job{
        Group: "group0",
        Name: "foo",
        Func: func() error {
            log.Println("I'm a job 2")
        },
    })

    // This job will be executed after all jobs in group0
    s.Add(scheduler.Job{
        Dependencies: []string{"group0"},
        Func: func() error {
            log.Println("I'm a job 3")
        },
    })

    // This job will be executed after 'foo' job in group0
    s.Add(scheduler.Job{
        Dependencies: []string{"group0:foo"},
        Func: func() error {
            log.Println("I'm a job 4")
        },
    })

    s.Execute()
}

Shortcuts

...

func main() {
    // All jobs will be executed in parallel
    scheduler.For(10, func(i int) error {
        log.Println("I'm a job", i)
    })

    // All jobs will be executed in parallel
    items := []interface{}{"foo", "bar", "baz"}
    scheduler.ForEach(items, func(item interface{}) error {
        log.Println("I'm a job with", item)
    })
}
Owner
Kyoto Framework
Set of libraries and tools, that becomes a framework
Kyoto Framework
Similar Resources

Tiny library to handle background jobs.

bgjob Tiny library to handle background jobs. Use PostgreSQL to organize job queues. Highly inspired by gue Features Durable job storage At-least-ones

Nov 16, 2021

A way of scheduling volcano jobs

JobFlow 背景 volcano Volcano是CNCF 下首个也是唯一的基于Kubernetes的容器批量计算平台,主要用于高性能计算场景。 它提供了Kubernetes目前缺 少的一套机制,这些机制通常是机器学习大数据应用、科学计算、 特效渲染等多种高性能工作负载所需的。 现状:当前vol

Oct 12, 2022

Executes jobs in separate GO routines. Provides Timeout, StartTime controls. Provides Cancel all running job before new job is run.

jobExecutor Library to execute jobs in GO routines. Provides for Job Timeout/Deadline (MaxDuration()) Job Start WallClock control (When()) Add a job b

Jan 10, 2022

Graceful shutdown with repeating "cron" jobs (running at a regular interval) in Go

Graceful shutdown with repeating "cron" jobs (running at a regular interval) in Go Illustrates how to implement the following in Go: run functions ("j

May 30, 2022

GPU Sharing Scheduler for Kubernetes Cluster

GPU Sharing Scheduler for Kubernetes Cluster

GPU Sharing Scheduler Extender in Kubernetes Overview More and more data scientists run their Nvidia GPU based inference tasks on Kubernetes. Some of

Jan 6, 2023

Package tasks is an easy to use in-process scheduler for recurring tasks in Go

Tasks Package tasks is an easy to use in-process scheduler for recurring tasks in Go. Tasks is focused on high frequency tasks that run quick, and oft

Dec 18, 2022

A lightweight job scheduler based on priority queue with timeout, retry, replica, context cancellation and easy semantics for job chaining. Build for golang web apps.

Table of Contents Introduction What is RIO? Concern An asynchronous job processor Easy management of these goroutines and chaining them Introduction W

Dec 9, 2022

A simple job scheduler backed by Postgres.

A simple job scheduler backed by Postgres used in production at https://operand.ai. Setup needs two environment variables, SECRET and ENDPOINT. The se

Sep 10, 2022

Chrono is a scheduler library that lets you run your task and code periodically

Chrono is a scheduler library that lets you run your task and code periodically

Chrono is a scheduler library that lets you run your tasks and code periodically. It provides different scheduling functionalities to make it easier t

Dec 26, 2022
Statefulset-scheduler (aka sfs-scheduler)

statefulset-scheduler (aka sfs-scheduler) Installation I already upload docker i

Dec 19, 2021
Scheduler: the scheduler of distbuild written in Golang

scheduler Introduction scheduler is the scheduler of distbuild written in Go. Pr

Feb 9, 2022
Scheduler - Scheduler package is a zero-dependency scheduling library for Go

Scheduler Scheduler package is a zero-dependency scheduling library for Go Insta

Jan 14, 2022
Linstor-scheduler-extender - LINSTOR scheduler extender plugin for Kubernetes

linstor-scheduler-extender LINSTOR scheduler extender plugin for Kubernetes whic

Dec 30, 2022
Crane scheduler is a Kubernetes scheduler which can schedule pod based on actual node load.

Crane-scheduler Overview Crane-scheduler is a collection of scheduler plugins based on scheduler framework, including: Dynamic scheuler: a load-aware

Dec 29, 2022
Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs
Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs

Jenkins Job DSL Plugin Introduction Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs. Unfortunately,

Dec 20, 2022
gron, Cron Jobs in Go.

gron Gron provides a clear syntax for writing and deploying cron jobs. Goals Minimalist APIs for scheduling jobs. Thread safety. Customizable Job Type

Dec 20, 2022
A persistent and flexible background jobs library for go.

Jobs Development Status Jobs is no longer being actively developed. I will still try my best to respond to issues and pull requests, but in general yo

Nov 21, 2022
Run Jobs on a schedule, supports fixed interval, timely, and cron-expression timers; Instrument your processes and expose metrics for each job.

A simple process manager that allows you to specify a Schedule that execute a Job based on a Timer. Schedule manage the state of this job allowing you to start/stop/restart in concurrent safe way. Schedule also instrument this Job and gather metrics and optionally expose them via uber-go/tally scope.

Dec 8, 2022
YTask is an asynchronous task queue for handling distributed jobs in golang
YTask is an asynchronous task queue for handling distributed jobs in golang

YTask is an asynchronous task queue for handling distributed jobs in golang

Dec 24, 2022