Use this template if you are reporting a bug. Don't use this template of you are proposing a feature.

Expected Behavior

When adding jobs, if they have not run yet, should not increase failed jobs count

Actual Behavior

For each new job added, it increases failed job count

Steps to Reproduce the Problem

1. Add a job via API
2. Failed job count increases

Specifications

• Version:v2.0.0-b5
• Platform:linux
• Backend store:default

Expected Behavior

Jobs should be executed on time without missing

Actual Behavior

Some of our jobs skip their execution once, and are not executed again, because the "next execution time" doesn't arrive (since it's in the past) So far it only happens to our jobs that their schedule intervals are longer than 6 hours(I believe its a coincidence but still won't hurt to mention)

Steps to Reproduce the Problem

1.create job and schedule it to run every day (@every 1440m) 2.check the next day if it ran

Specifications

• Version:2.0.0-rc3
• Platform:linux
• Backend store: default boltDB

Hey,

I migrate from command (since declared as deprecated) to shell executor and I got the following issue from the agent :

INFO[2018-05-23T19:08:20Z] agent: Starting job                           job=stats_update node=04fe58fe3cf0
ERRO[2018-05-23T19:08:20Z] invoke: Specified executor is not present     executor="<nil>" node=04fe58fe3cf0

with following config :

{
    "name": "stats_update",
    "schedule": "@every 15s",
    "executor": "shell",
    "executor_config": {
        "command": "./stats_update",
        "shell": false
    },
    "concurrency": "forbid",
    "tags": {
        "role": "manager:1"
    }
}

I'm on 0.10.1 dkron version.

My config is bad or it's bug ?

Hey. I've got a bunch of instances running dkron with a Consul cluster as their backing store. I'm trying to update from Dkron 0.7 to 0.9.1. The instances that are running 0.9.1 throw this error:

time="2016-10-25T14:52:07-04:00" level=info msg="agent: Listen for events" node=ip-10-0-15-163
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x48897c]

goroutine 50 [running]:
panic(0xbb9dc0, 0xc820010060)
        /Users/victorcoder/src/github.com/mxcl/homebrew/Cellar/go/1.6.3/libexec/src/runtime/panic.go:481 +0x3e6
github.com/victorcoder/dkron/dkron.(*AgentCommand).eventLoop(0xc82000c4e0)
        /Users/victorcoder/src/github.com/victorcoder/dkron/dkron/agent.go:490 +0x1abc
created by github.com/victorcoder/dkron/dkron.(*AgentCommand).Run
        /Users/victorcoder/src/github.com/victorcoder/dkron/dkron/agent.go:298 +0x362
Starting Dkron agent...
time="2016-10-25T14:52:07-04:00" level=info msg="agent: Dkron agent starting" node=ip-10-0-15-163

I'm not doing anything crazy here, I don't think. It's been working for months. Any ideas?

More debug:

// /etc/dkron/dkron.json
{
  "backend": "consul",
  "backend_machine": "consul.internal:8500",
  "keyspace": "bkg-process",
  "http_addr": ":8988",
  "server": true,
  "encrypt": "KEY",
  "tags": {
    "role": "background_processing"
  },
  "join": "background-processing.service.dc1.consul"
}

$ curl 127.0.0.1:8988/v1/jobs
[{"name":"cron_job","schedule":"0 * * * * *","shell":false,"command":"/queue.sh","owner":"Me,"owner_email":"[email protected]","success_count":254066,"error_count":0,"last_success":"2016-10-25T15:00:00.23316342-04:00","last_error":"0001-01-01T00:00:00Z","disabled":false,"tags":{"role":"background_processing:1"},"retries":0,"dependent_jobs":null,"parent_job":""}]

Originally posted to #377

Hi @victorcoder

I'm on 0.11.2 and have noticed that the concurrency: forbid is not always respected. I have a very long running process (nearly 2 hours) that I poll every 15 minutes. It is essential that it only one instance is running.

I use Consul as the backend, so thought that might be my issue, but I just completed a migration of the Consul cluster, and the Dkron masters, to greatly reduce the node count, and the latency between the nodes. But dkron seems to erroneously mark an event as complete (erroneously, because I can still see the task is running via ps aux) and fires again at the next schedule. If I forcibly kill the event that was marked complete, it gets updated with the new completed timestamp.

I cannot reliably reproduce the issue, as it seems random. Please let me know what I can do to help track down the cause.

Geoff

My cluster consists of 3 server node (Dkron 3.1.7):

[opc@nonprod-exa-services-0 ~]$ dkron version
Name: Dkron
Version: 3.1.7
Codename: merichuas
Agent Protocol: 5 (Understands back to: 2)

All nodes are configured as follows:

[opc@nonprod-exa-services-0 ~]$ cat /etc/dkron/dkron.yml
server: true
bootstrap-expect: 3
join:
- nonprod-exa-services-0.nonprodexasvc.nonprodvcn.oraclevcn.com
- nonprod-exa-services-1.nonprodexasvc.nonprodvcn.oraclevcn.com
- nonprod-exa-services-2.nonprodexasvc.nonprodvcn.oraclevcn.com
log-level: debug

This cluster has been running for ~3 weeks and able to execute scheduled jobs

[opc@nonprod-exa-services-0 ~]$ systemctl status dkron
● dkron.service - Dkron Agent
   Loaded: loaded (/usr/lib/systemd/system/dkron.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-06-02 10:19:43 CEST; 3 weeks 1 days ago
     Docs: https://dkron.io
 Main PID: 6940 (dkron)
    Tasks: 95
   Memory: 202.0M
   CGroup: /system.slice/dkron.service
           ├─6940 /usr/bin/dkron agent
           ├─6945 /usr/bin/dkron-processor-files
           ├─6952 /usr/bin/dkron-processor-fluent
           ├─7049 /usr/bin/dkron-processor-log
           ├─7056 /usr/bin/dkron-processor-syslog
           ├─7062 /usr/bin/dkron-executor-gcppubsub
           ├─7069 /usr/bin/dkron-executor-http
           ├─7076 /usr/bin/dkron-executor-kafka
           ├─7083 /usr/bin/dkron-executor-nats
           ├─7088 /usr/bin/dkron-executor-rabbitmq
           └─7096 /usr/bin/dkron-executor-shell

ISSUE Details:

One of the nodes is unable to contact the other 2 nodes (could be an intermittent network issue) causing a re-election. After re-election none of the scheduled jobs are executing

Log snippet from one of the nodes. The complete logs from the 3 nodes are attached.

o msg="2021/06/23 23:28:42 [DEBUG] memberlist: Initiating push/pull sync with: nonprod-exa-services-1 10.106.51.22:8946"
o msg="2021-06-23T23:28:44.338+0200 [WARN]  raft: failed to contact: server-id=nonprod-exa-services-0 time=500.962849ms"
o msg="2021-06-23T23:28:44.390+0200 [WARN]  raft: failed to contact: server-id=nonprod-exa-services-1 time=500.680357ms"
o msg="2021-06-23T23:28:44.390+0200 [WARN]  raft: failed to contact: server-id=nonprod-exa-services-0 time=552.570599ms"
o msg="2021-06-23T23:28:44.390+0200 [WARN]  raft: failed to contact quorum of nodes, stepping down"
o msg="2021-06-23T23:28:44.390+0200 [INFO]  raft: entering follower state: follower=\"Node at 10.106.51.29:6868 [Follower]\" leader="
o msg="2021-06-23T23:28:44.390+0200 [INFO]  raft: aborting pipeline replication: peer=\"{Voter nonprod-exa-services-1 10.106.51.22:6868}\""
o msg="2021-06-23T23:28:44.390+0200 [INFO]  raft: aborting pipeline replication: peer=\"{Voter nonprod-exa-services-0 10.106.51.18:6868}\""
ug msg="dkron: shutting down leader loop" node=nonprod-exa-services-2
ug msg="scheduler: Stopping scheduler" node=nonprod-exa-services-2
o msg="dkron: cluster leadership lost" node=nonprod-exa-services-2
o msg="dkron: monitoring leadership" node=nonprod-exa-services-2
o msg="2021-06-23T23:28:46.085+0200 [WARN]  raft: heartbeat timeout reached, starting election: last-leader="
o msg="2021-06-23T23:28:46.085+0200 [INFO]  raft: entering candidate state: node=\"Node at 10.106.51.29:6868 [Candidate]\" term=54"
o msg="2021-06-23T23:28:46.092+0200 [DEBUG] raft: votes: needed=2"
o msg="2021-06-23T23:28:46.092+0200 [DEBUG] raft: vote granted: from=nonprod-exa-services-2 term=54 tally=1"
o msg="2021-06-23T23:28:46.093+0200 [DEBUG] raft: vote granted: from=nonprod-exa-services-1 term=54 tally=2"
o msg="2021-06-23T23:28:46.093+0200 [INFO]  raft: election won: tally=2"
o msg="2021-06-23T23:28:46.094+0200 [INFO]  raft: entering leader state: leader=\"Node at 10.106.51.29:6868 [Leader]\""
o msg="2021-06-23T23:28:46.094+0200 [INFO]  raft: added peer, starting replication: peer=nonprod-exa-services-1"
o msg="2021-06-23T23:28:46.094+0200 [INFO]  raft: added peer, starting replication: peer=nonprod-exa-services-0"
o msg="dkron: cluster leadership acquired" node=nonprod-exa-services-2
o msg="dkron: monitoring leadership" node=nonprod-exa-services-2
o msg="2021-06-23T23:28:46.095+0200 [INFO]  raft: pipelining replication: peer=\"{Voter nonprod-exa-services-1 10.106.51.22:6868}\""
o msg="2021-06-23T23:28:46.097+0200 [INFO]  raft: pipelining replication: peer=\"{Voter nonprod-exa-services-0 10.106.51.18:6868}\""
o msg="agent: Starting scheduler" node=nonprod-exa-services-2

Some time after start, all nodes of my cluster (two servers and two agents) crash with the same error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x160 pc=0x948a8a]

goroutine 623378 [running]:
google.golang.org/grpc.(*Server).Stop(0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1482 +0x4a
github.com/distribworks/dkron/v3/plugin.(*ExecutorClient).Execute(0xc00091e6a0, 0xc00b231640, 0x357c300, 0xc000e27820, 0xc0011bc980, 0x25bb801, 0xc0044c6150)
        /home/runner/work/dkron/dkron/plugin/executor.go:65 +0x190
github.com/distribworks/dkron/v3/dkron.(*AgentServer).AgentRun(0xc0001295b8, 0xc0044c6060, 0x35e2620, 0xc003374e20, 0x0, 0x0)
        /home/runner/work/dkron/dkron/dkron/grpc_agent.go:83 +0x6f4
github.com/distribworks/dkron/v3/plugin/types._Agent_AgentRun_Handler(0x2561d80, 0xc0001295b8, 0x35dcda0, 0xc0002186c0, 0x4ba5ae0, 0xc000e2a200)
        /home/runner/work/dkron/dkron/plugin/types/dkron.pb.go:1862 +0x109
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000a12b60, 0x35ea7e0, 0xc0019b4300, 0xc000e2a200, 0xc000b56150, 0x4b45e20, 0x0, 0x0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1329 +0xcbc
google.golang.org/grpc.(*Server).handleStream(0xc000a12b60, 0x35ea7e0, 0xc0019b4300, 0xc000e2a200, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1409 +0xc64
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001ab37e0, 0xc000a12b60, 0x35ea7e0, 0xc0019b4300, 0xc000e2a200)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:746 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:744 +0xa1
runtime: note: your Linux kernel may be buggy
runtime: note: see https://golang.org/wiki/LinuxKernelSignalVectorBug
runtime: note: mlock workaround for kernel bug failed with errno 12

Environment

• Dkron 3.0.5

• Docker (servers: Dockerfile, run script, agents: Dockerfile, run script)

• uname -a:

    Linux dkron-server-1.backpack.test 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  

1. A job is not picked for the next run and so miss all subsequent runs.
2. Also when a job is run manually using the play button on the dkron dashboard the next run time it picks is the current time (I have been using @every construct of the cronspec).
3. On using the API doc to create new jobs, the first run is executed successfully but next run time is not updated to so all subsequent runs fail.

Steps to reproduce the behavior:

1. Create a new job using @every construct of the cronspec, second and the following runs fail.
2. Use the play button on the dashboard. Next run time will be picked as the current time.

• OS: centos
• Version 2 rc9

I would like to know if there is a way to group jobs, like a custom tag or something. My requirement is that I need to create several jobs for an user, and at some point I need to delete all jobs which were not executed. I tried to do it using "tags" (tags: { userId: "123" }) but the http executor does not run, although the job is created.

Hello guys,

let me introduce this little change which could be useful for somebody else.

From now on we can assign labels to the jobs. And query (GET /jobs) them with labels filter. This change could be useful for the users who have to build UI and display background jobs on their own. One of the use case when you have a different kind of jobs like the system jobs and some application related and you don't want the app to see the jobs not assigned to it.

Exactly our case is that we have 3 types of jobs and 3 different UIs to display the jobs and their status. All jobs should now know about each other. Moreover, we have that 3 UI build for 12 countries. Which lead to 36 UIs. Before that change, we used prefixes like country-app_name-actual_name_of_the_cron and it's getting annoying now.

Feel free to comment and accept ;)

Good luck

UPDATE: 2018-08-29

No more labels, use tags to filter jobs.

Describe the bug Hi, There's a one issue that I can't pinpoint and need the help. We have around 50 jobs that are being scheduled at different times and we're using master version. Sometimes, by no reason (so far) random jobs at random times are not executed, despite visible they're scheduled. I've modified logs a bit to see a flow of a job from time being scheduled to end of execution. During the issue the flow is the following:

time="2020-04-21T02:00:00Z" level=debug msg="scheduler: Run job" job=sync_local_ip_list_dms_0 node=dkron-server-1 schedule="0 0 2 * * *"
time="2020-04-21T02:00:00Z" level=debug msg="store: Retrieved job from datastore" job=sync_local_ip_list_dms_0 node=dkron-server-1
time="2020-04-21T02:00:00Z" level=info msg="agent: Sending query" job_name=sync_local_ip_list_dms_0 node=dkron-server-1 query="run:job"
time="2020-04-21T02:00:00Z" level=debug msg="store: Setting job" job=sync_local_ip_list_dms_0 node=dkron-server-1
time="2020-04-21T02:00:00Z" level=debug msg="store: Retrieved job from datastore" job=sync_local_ip_list_dms_0 node=dkron-server-1
time="2020-04-21T02:00:00Z" level=debug msg="agent: Filtered nodes to run" job_name=sync_local_ip_list_dms_0 job_name_from_function=sync_local_ip_list_dms_0 node=dkron-server-1 nodes="[dms-dkron-kapp-5dbf554656-vr99b]"
time="2020-04-21T02:00:00Z" level=debug msg="agent: Filtered tags to run" job_name=sync_local_ip_list_dms_0 job_name_from_function=sync_local_ip_list_dms_0 node=dkron-server-1 tags="map[dms:cron region:global]"
time="2020-04-21T02:00:00Z" level=debug msg="agent: Sending query" job_name=sync_local_ip_list_dms_0 json="{\"execution\":{\"job_name\":\"sync_local_ip_list_dms_0\",\"started_at\":\"0001-01-01T00:00:00Z\",\"finished_at\":\"0001-01-01T00:00:00Z\",\"success\":false,\"group\":1587434400017697229,\"attempt\":1},\"rpc_addr\":\"172.25.4.133:6868\"}" node=dkron-server-1 query="run:job"
time="2020-04-21T02:00:08Z" level=debug msg="agent: Done receiving acks and responses" fields.time=8.014626936s job=sync_local_ip_list_dms_0 node=dkron-server-1 query="run:job"

In normal cases, before "Done receiving acks and responses" step, there are also steps: "agent: Received ack" and "agent: Received response"

It looks like chosen agent node in step "Filtered nodes to run" by some reason is ignoring or not receiving broadcasted query.

From the view of agent node, there is simply no log message "Running job" for a given job. Though there are visible log messages from serf like

time="2020-04-21T02:00:00Z" level=info msg="2020/04/21 02:00:00 [DEBUG] serf: messageQueryType: run:job"

Due to an intense workload, it's hard to tell if that log message is generated from a missed job.

What can we do about this? To pinpoint the problem and fix it.

Expected behavior No misses!

Bumps github.com/nats-io/nats.go from 1.21.0 to 1.22.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps github.com/gin-gonic/gin from 1.8.1 to 1.8.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Describe the bug When trying to bring up a simple cluster (1 server/1 agent) on our production docker swarm cluster the agent can never connect to the server; it even tries connecting to the wrong IP address. Doing the exact same thing on our standalone test box (minus the deploy restrictions in the compose file) works flawlessly. I even added a placement restriction for the swam to force them to the same node and it doesn't work.

To Reproduce Production swarm stack file:

` version: '3.8'

services: dkron-server: image: centricdocker.azurecr.io/thirdparty/dkron ports: - "10120:8080"
volumes: - /var/apps/docker-shared/shared-dkron/root:/root - /media/scripts:/media/scripts - /var/apps/docker-shared/shared-dkron/dkron-server/data:/data command: dkron agent --server --log-level=debug --bootstrap-expect=1 --data-dir=/data --node-name dkron-primary stop_grace_period: 3m deploy: replicas: 1 placement: constraints: - node.labels.dkron==allowed

dkron-agent: image: centricdocker.azurecr.io/thirdparty/dkron depends_on: - dkron-server volumes: - /var/apps/docker-shared/shared-dkron/root:/root - /media/scripts:/media/scripts command: dkron agent --retry-join=dkron-server:8946 --log-level=debug --tag agent=true stop_grace_period: 3m deploy: replicas: 1 placement: constraints: - node.labels.dkron==allowed ` Does not work.

Standalone server stack file: ` version: '3.8'

services: dkron-server: image: centricdocker.azurecr.io/thirdparty/dkron ports: - "10120:8080"
volumes: - /var/apps/docker-shared/shared-dkron/root:/root - /media/scripts:/media/scripts - /var/apps/docker-shared/shared-dkron/dkron-server/data:/data command: dkron agent --server --log-level=debug --bootstrap-expect=1 --data-dir=/data --node-name dkron-primary stop_grace_period: 3m

dkron-agent: image: centricdocker.azurecr.io/thirdparty/dkron depends_on: - dkron-server volumes: - /var/apps/docker-shared/shared-dkron/root:/root - /media/scripts:/media/scripts command: dkron agent --retry-join=dkron-server:8946 --log-level=debug --tag agent=true stop_grace_period: 3m ` This works perfectly.

Expected behavior The swarm should work and the agent should try to connect to the correct internal IP of 10.0.61.9, but it is trying to connect to 10.0.61.2 instead. The server/web-ui is up.

Screenshots

** Specifications:**

• OS: Linux
• VersionLatest

Bumps github.com/hashicorp/go-plugin from 1.4.5 to 1.4.8.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps github.com/prometheus/client_golang from 1.13.0 to 1.14.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps github.com/spf13/viper from 1.13.0 to 1.14.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.