A way of scheduling volcano jobs

JobFlow

背景

volcano

Volcano是CNCF 下首个也是唯一的基于Kubernetes的容器批量计算平台,主要用于高性能计算场景。 它提供了Kubernetes目前缺 少的一套机制,这些机制通常是机器学习大数据应用、科学计算、 特效渲染等多种高性能工作负载所需的。

现状:当前volcano开源社区对于complete依赖的作业的解决方案是引入argo workflow。

  1. 但目前volcano中缺乏对vcjob的编排,缺乏vcjob之间的作业依赖运行实现。当出现complete依赖的task时则与gang调度算法相矛盾。 虽然argo workflow也能很好的支持vcjob的编排,argo workflow的引入又太过庞大。

  2. argo workflow更适用于广泛的场景,并没有针对某个场景的特定情况有细分的需求,例如使用argo workflow创建vcjob的任务依赖无法直接查看个vcjob的具体情况。 JobFlow根据需求场景适配vcjob。更贴合vcjob的使用。

  3. workflow目前只针对vcjob completed状态的场景有支持。 jobflow 可以针对vcjob增加对task running状态以及探针方式的依赖等等,依赖方式多样化。

  4. workflow目前可观察到下发的vcjob级别的进度。但当vcjob支持task级别的任务依赖后,workflow无法查看task级别的进度。 JobFlow则可提供该场景。

字段定义

JobTemplate

jobtemplate简写 (jt) ,可通过 kubectl get jt 查看该资源

希望可以生成vcctl直接转换template和vcjob

jobtemplate和vcjob可以互相转换,区别是jobtemplate不会被job controller下发,jobflow可以直接引用该JobTemplate名称,来实现vcjob的下发。

apiVersion: batch.volcano.sh/v1alpha1
kind: JobTemplate
metadata:
  name: A
spec:
  minAvailable: 3
  schedulerName: volcano
  priorityClassName: high-priority
  policies:
    - event: PodEvicted
      action: RestartJob
  plugins:
    ssh: []
    env: []
    svc: []
  maxRetry: 5
  queue: default
  volumes:
    - mountPath: "/myinput"
    - mountPath: "/myoutput"
      volumeClaimName: "testvolumeclaimname"
      volumeClaim:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "my-storage-class"
        resources:
          requests:
            storage: 1Gi
  tasks:
    - replicas: 6
      name: "default-nginx"
      template:
        metadata:
          name: web
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "1"
          restartPolicy: OnFailure
---
apiVersion: batch.volcano.sh/v1alpha1
kind: JobTemplate
metadata:
  name: B
spec:
  minAvailable: 3
  schedulerName: volcano
  priorityClassName: high-priority
  policies:
    - event: PodEvicted
      action: RestartJob
  plugins:
    ssh: []
    env: []
    svc: []
  maxRetry: 5
  queue: default
  volumes:
    - mountPath: "/myinput"
    - mountPath: "/myoutput"
      volumeClaimName: "testvolumeclaimname"
      volumeClaim:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "my-storage-class"
        resources:
          requests:
            storage: 1Gi
  tasks:
    - replicas: 6
      name: "default-nginx"
      template:
        metadata:
          name: web
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "1"
          restartPolicy: OnFailure
---
apiVersion: batch.volcano.sh/v1alpha1
kind: JobTemplate
metadata:
  name: C
spec:
  minAvailable: 3
  schedulerName: volcano
  priorityClassName: high-priority
  policies:
    - event: PodEvicted
      action: RestartJob
  plugins:
    ssh: []
    env: []
    svc: []
  maxRetry: 5
  queue: default
  volumes:
    - mountPath: "/myinput"
    - mountPath: "/myoutput"
      volumeClaimName: "testvolumeclaimname"
      volumeClaim:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "my-storage-class"
        resources:
          requests:
            storage: 1Gi
  tasks:
    - replicas: 6
      name: "default-nginx"
      template:
        metadata:
          name: web
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "1"
          restartPolicy: OnFailure
---
apiVersion: batch.volcano.sh/v1alpha1
kind: JobTemplate
metadata:
  name: D
spec:
  minAvailable: 3
  schedulerName: volcano
  priorityClassName: high-priority
  policies:
    - event: PodEvicted
      action: RestartJob
  plugins:
    ssh: []
    env: []
    svc: []
  maxRetry: 5
  queue: default
  volumes:
    - mountPath: "/myinput"
    - mountPath: "/myoutput"
      volumeClaimName: "testvolumeclaimname"
      volumeClaim:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "my-storage-class"
        resources:
          requests:
            storage: 1Gi
  tasks:
    - replicas: 6
      name: "default-nginx"
      template:
        metadata:
          name: web
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "1"
          restartPolicy: OnFailure
---
apiVersion: batch.volcano.sh/v1alpha1
kind: JobTemplate
metadata:
  name: E
spec:
  minAvailable: 3
  schedulerName: volcano
  priorityClassName: high-priority
  policies:
    - event: PodEvicted
      action: RestartJob
  plugins:
    ssh: []
    env: []
    svc: []
  maxRetry: 5
  queue: default
  volumes:
    - mountPath: "/myinput"
    - mountPath: "/myoutput"
      volumeClaimName: "testvolumeclaimname"
      volumeClaim:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "my-storage-class"
        resources:
          requests:
            storage: 1Gi
  tasks:
    - replicas: 6
      name: "default-nginx"
      template:
        metadata:
          name: web
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "1"
          restartPolicy: OnFailure

JobFlow

jobflow简写 (jf) ,可通过 kubectl get jf 查看该资源

jobflow旨在实现volcano中vcjob之间的作业依赖运行。根据vcjob之间的依赖关系对vcjob进行下发。

apiVersion: batch.volcano.sh/v1alpha1
kind: JobFlow
metadata:
  name: test
  namespace: default
  creationTimestamp: "2021-10-18T07:01:24Z"
spec:
  JobRetainPolicy: retain/delete   jobflow运行结束后,保留产生的job. 否则删除。
  flows:
    - name: A
    - name: B
      dependsOn:
        targets: [‘A’]
    - name: C
      dependsOn:
        targets: [‘B’]
    - name: D
      dependsOn:
        targets: [‘B’]
    - name: E
      dependsOn:
        targets: [‘C’,‘D’]
status:
  jobStatusList: []
  pendingJobs: []
  runningJobs: []
  failedJobs: []
  completedJobs: []
  terminatedJobs: []
  unKnowJobs: []
  conditions: 
  - A: {
    phase: completed
    message: ''
    createTime: "2021-08-25T02:04:20Z"
    RunningDuration: "22s"
    taskStatusCount: 
    - nginx:  
      - phase:
        - Running: 1
    }
  - B: {
      phsae: running
      message: ''
      createTime: "2021-08-25T02:04:20Z"
      RunningDuration: "",
     }
  - C: {
      phsae: waiting
      message: ''
      createTime: ""
      RunningDuration: ""
     }
  - D: {
      phsae: waiting
      message: ''
      createTime: ""
      RunningDuration: ""
      }
  - E: {
      phsae: waiting
      message: ''
      createTime: ""
      RunningDuration: ""
      }
  state:
    phase: successed/terminating/terminated/failed/pending

jobStatusList

type JobStatus struct {
    Name string
    State string // running/failed
    StartTimestamp Time
    EndTimestamp Time
    RestartCount int
    RunningHistories []JobRunningHistory
}

type JobRunningHistory struct {
    StartTimestamp Time
    EndTimestamp time
    State string // failded/succeeded ....
}

根据JobFlow创建的名称遵循JobFlowName-JobTemplateName,

JobFlow解释

  • pendingJobs 处于pending状态的vcjob
  • runningJobs 处于running状态的vcjob
  • failedJobs 处于failedJobs状态的vcjob
  • completed 处于completed状态的vcjob
  • terminated 处于terminated状态的vcjob
  • metadata: 描述JobFlow的元数据信息
  • flow:定义了vcjob之间的依赖关系,没有依赖项的vcjob即位入口。支持多入口和多出口。当前只支持了complete依赖
  • depends.target: 指定了依赖的vcjob
  • jobStatusList 拆分出来的所有vcjob的状态信息
  • successfulJobList 已经成功complete的vcjob
  • conditions 用于描述所有vcjob当前的状态,创建时间,完成时间以及信息,该处的vcjob状态额外增加waiting状态用于描述依赖项没有达到要求的vcjob。
  • phase JobFlow的状态
Succeed: 所有的vcjob都已达到completed状态。
Terminating:jobflow正在删除。
Failed: flow中某个vcjob处于failed状态导致flow中的vcjob无法继续下发。
Running: flow中包含处于Running状态vcjob。
Pending: flow中包不含处于Running状态vcjob。

JobFlow webhook 校验

创建JobFlow校验
1、JobFlow依赖关系中不能有同名template
  如:A->B->A->C  A出现了两次
2、JobFlow中不能出现闭环
  如:A -> B  ->  C
          ^     /
          |    /
          < - D

创建JobTemplte校验(遵循vcjob参数规范)
如:job minAvailable 必须大于等于零
   job maxRetry 必须大于等于零
   tasks不能为空,不能有同名task
   task副本数不能小于零
   task minAvailable不能大于task replicas...

JobFlow controller

  1. 根据namespace加载JobFlow 资源。
  2. 根据namespace加载jobflow。
  3. 下发入口的vcjob。
  4. 根据vcjob依赖下发vcjob。遍历判断非入口的vcjob的依赖是否满足条件,若满足则进行下发。
  5. 将当前的状态信息写入jobflow的status。
  6. 监听jobflow下发的vcjob。

开发

通过make generateDeployYaml 生成部署文件到deploy目录下

需要更改yaml文件时需要更改kustomize相关配置模板和patch文件,最后通过make generateDeployYaml生成对应yaml文件。不建议直接修改deploy下的yaml文件

演示视频

https://www.bilibili.com/video/BV1c44y1Y7FX

部署

make build   #生成二进制执行文件./bin/manager

make docker-build     #构建镜像

kubectl apply -f ./deploy/     #部署

Similar Resources

Simple, zero-dependency scheduling library for Go

go-quartz Simple, zero-dependency scheduling library for Go. About Inspired by the Quartz Java scheduler. Library building blocks Job interface. Any t

Dec 30, 2022

Easy and fluent Go cron scheduling

goCron: A Golang Job Scheduling Package. goCron is a Golang job scheduling package which lets you run Go functions periodically at pre-determined inte

Jan 8, 2023

Job scheduling made easy.

scheduler Job scheduling made easy. Scheduler allows you to schedule recurrent jobs with an easy-to-read syntax. Inspired by the article Rethinking Cr

Dec 30, 2022

goCron: A Golang Job Scheduling Package.

goCron: A Golang Job Scheduling Package.

Jan 9, 2023

Distributed Task Scheduling System|分布式定时任务调度平台

Distributed Task Scheduling System|分布式定时任务调度平台

Crocodile Distributed Task Scheduling System English | 中文 Introduction A distributed task scheduling system based on Golang that supports http request

Jan 5, 2023

nano-gpu-scheduler is a Kubernetes scheduler extender for GPU resources scheduling.

nano-gpu-scheduler is a Kubernetes scheduler extender for GPU resources scheduling.

Nano GPU Scheduler About This Project With the continuous evolution of cloud native AI scenarios, more and more users run AI tasks on Kubernetes, whic

Dec 29, 2022

K8s cluster simulator for workload scheduling.

K8s cluster simulator for workload scheduling.

Open-Simulator Motivation 概念定义 Open-Simulator 是 K8s 下的仿真调度组件。用户准备一批待创建 Workload 资源,Workload 资源指定好资源配额、绑核规则、亲和性规则、优先级等,通过 Open-Simulator 的仿真调度能力可判断当前集群

Dec 25, 2022

A zero-dependencies and lightweight go library for job scheduling

A zero-dependencies and lightweight go library for job scheduling.

Aug 3, 2022

Scheduler - Scheduler package is a zero-dependency scheduling library for Go

Scheduler Scheduler package is a zero-dependency scheduling library for Go Insta

Jan 14, 2022
Comments
  • How to use jobflow to depend on the status of a job in the previous task?

    How to use jobflow to depend on the status of a job in the previous task?

    Hi, I used jobflow to build a tensorflow-ps-workers task, and when the tensorflow distributed training finished. The ps-job will be terminated and the workers were succeed. But the jobflow can not start the next job which depends on the previous whole task job. It looks like the jobflow was affected by the state of ps-job. So, I want to know how to start the next job with the one of the whole task(such as the completed worker-job)? Hope to reply soon, thanks!

  • 创建jobflow无法正常运行

    创建jobflow无法正常运行

    你好,我在搭建好的集群尝试使用jobflow jobtemplate.yaml如下:

    apiVersion: batch.volcano.sh/v1alpha1
    kind: JobTemplate
    metadata:
      name: a
    spec:
      minAvailable: 1
      schedulerName: volcano
      queue: default
      policies:
        - event: PodEvicted
          action: RestartJob
      maxRetry: 5
      queue: default
    
      tasks:
        - replicas: 1
          name: "test-task"
          policies: 
            - event: TaskCompleted
              action: CompletedJob
          template:
            metadata:
              name: test-task
            spec:
              containers:
                - name: test
                  image: centos
                  imagePullPolicy: IfNotPresent
                  command: ["/bin/bash", "-c"]
                  args: ["touch /home/a.txt && echo 'this is test string' >> /home/a.txt && sleep 10 "]
                  resources:
                    requests:
                      cpu: "1"
                  volumeMounts:
                    - mountPath: /home
                      name: my-pvc
              volumes:
              - name: my-pvc
                persistentVolumeClaim:
                  claimName: my-pvc-test
    ---
    apiVersion: batch.volcano.sh/v1alpha1
    kind: JobTemplate
    metadata:
      name: b
    spec:
      minAvailable: 1
      schedulerName: volcano
      queue: default
      policies:
        - event: PodEvicted
          action: RestartJob
      maxRetry: 5
      queue: default
    
      tasks:
        - replicas: 1
          name: "test-task"
          policies: 
            - event: TaskCompleted
              action: CompletedJob
          template:
            metadata:
              name: test-task
            spec:
              containers:
                - name: test
                  image: centos
                  imagePullPolicy: IfNotPresent
                  command: ["/bin/bash", "-c"]
                  args: ["touch /home/b.txt && echo 'this is test string' >> /home/b.txt && sleep 10 "]
                  resources:
                    requests:
                      cpu: "1"
                  volumeMounts:
                    - mountPath: /home
                      name: my-pvc
              volumes:
              - name: my-pvc
                persistentVolumeClaim:
                  claimName: my-pvc-test
    ---
    

    job flow.yaml如下:

    apiVersion: batch.volcano.sh/v1alpha1
    kind: JobFlow
    metadata:
      name: test
      namespace: default
    spec:
      jobRetainPolicy: retain   # After jobflow runs, keep the generated job. Otherwise, delete it.
      flows:
        - name: a
        - name: b
          dependsOn:
            targets: ['a']
    

    我现在面临的问题是,create jobtemplate后create jobflow,执行 kubectl get jf,存在的jf无状态信息显示,如下:

    NAME   STATUS   AGE
    test            10m
    

    执行kubectl get pods ,也未见pod被创建。

gron, Cron Jobs in Go.

gron Gron provides a clear syntax for writing and deploying cron jobs. Goals Minimalist APIs for scheduling jobs. Thread safety. Customizable Job Type

Dec 20, 2022
A persistent and flexible background jobs library for go.

Jobs Development Status Jobs is no longer being actively developed. I will still try my best to respond to issues and pull requests, but in general yo

Nov 21, 2022
Run Jobs on a schedule, supports fixed interval, timely, and cron-expression timers; Instrument your processes and expose metrics for each job.

A simple process manager that allows you to specify a Schedule that execute a Job based on a Timer. Schedule manage the state of this job allowing you to start/stop/restart in concurrent safe way. Schedule also instrument this Job and gather metrics and optionally expose them via uber-go/tally scope.

Dec 8, 2022
YTask is an asynchronous task queue for handling distributed jobs in golang
YTask is an asynchronous task queue for handling distributed jobs in golang

YTask is an asynchronous task queue for handling distributed jobs in golang

Dec 24, 2022
Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs
Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs

Jenkins Job DSL Plugin Introduction Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs. Unfortunately,

Dec 20, 2022
Tiny library to handle background jobs.

bgjob Tiny library to handle background jobs. Use PostgreSQL to organize job queues. Highly inspired by gue Features Durable job storage At-least-ones

Nov 16, 2021
Scheduler: Go jobs execution system

Scheduler Go jobs execution system. Inspired by CI/CD and Unity task scheduler.

Jul 1, 2022
Executes jobs in separate GO routines. Provides Timeout, StartTime controls. Provides Cancel all running job before new job is run.

jobExecutor Library to execute jobs in GO routines. Provides for Job Timeout/Deadline (MaxDuration()) Job Start WallClock control (When()) Add a job b

Jan 10, 2022
Graceful shutdown with repeating "cron" jobs (running at a regular interval) in Go

Graceful shutdown with repeating "cron" jobs (running at a regular interval) in Go Illustrates how to implement the following in Go: run functions ("j

May 30, 2022
clockwork - Simple and intuitive job scheduling library in Go.
clockwork - Simple and intuitive job scheduling library in Go.

clockwork A simple and intuitive scheduling library in Go. Inspired by python's schedule and ruby's clockwork libraries. Example use package main imp

Jul 27, 2022