painless task queue manager for shell commands with an intuitive cli interface (execute shell commands in distributed cloud-native queue manager).

EXEQ

DOCS STILL IN PROGRESS.
Execute shell commands in queues via cli or http interface.

Features

  • Simple intuitive tiny cli app.
  • Modular queue backends (currently it supports redis as backend but we should support more in the future like sqs, kafka, postgres, ...).
  • Powerful configurations thanks to HCL by hashicrop.
  • OpenMetrics /metrics endpoint to inspect the queue via promethues.
  • Error reporting via sentry and stdout/stderr logging.
  • Easily create shortcuts for repeated shell commands (we name it Macros).
  • Limit the job execution time.

Components

  • the configuration file which uses hcl as a configuration language.
  • the binary itself which you can get from the releases page.

Config

value. // here we cat the data from hello.txt which we mounted before, then echo the value from // the arguments map exists in a key called message (just like $args['message']). command = "cat ./hello.txt && echo {{.Args.message}}" // after how long time should we kill the job? // a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, // such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". // empty means "no time limit". max_execution_time = "" // sometimes when you run this service within a cloud engine like k8s or docker container // you will have to define multiple configmaps/volumes mounts, // to simplify this job, we can only mount exeq configurations file and add any other file required by the job // into the following mounts block. mount "./hello.txt" { content = "hello world" } } ">
// here we define the logging related configs.
// kindly note that this config file is preprocessed at first with environment expander
// this means that you can easily use any ENV var in any value here i.e: ${HOME}
logging {
    // available level names are: "disable" "fatal" "error" "warn" "info" "debug".
    // NOTE: the value is case sensitive.
    log_level = "debug"

    // sentry dsn (for error reporting).
    // see https://sentry.io/
    // you can do this: sentry_dsn = ${SENTRY_DSN}, but you have to export that env
    // before running exeq.
    sentry_dsn = ""
}

// http server related configs.
http {
    // the address to start listening on.
    listen = ":1215"

    // whether to enable/disable access logs.
    access_logs = true
}

// queue related configs.
queue {
    // the driver used as queue backend.
    // the available drivers are: [rmq].
    driver = "rmq"

    // the data source name or the connection string for the selected driver.
    dsn = "redis://localhost:6379/1"

    // queue workers count.
    workers_count = 5

    // the duration between each poll from the queue
    // a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, 
    // such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
    poll_duration = "1s"

    // how many times a should we retry a failed job?
    retry_attempts = 3

    // each driver needs to keep jobs history, but you may not have to keep it for ever
    // so this history block helps you to configure the history feature.
    history {
        // for how long should we keep our history?
        // a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, 
        // such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
        retention_period = "48h"
    }
}

// here we define a macro which is considered an alias for a command.
// the macro name is "example1".
macro "example1" {
    // the command you want to execute when you call this macro.
    // you can pass arguments to the command and start using it in the command
    // you have to take a look at this first: https://pkg.go.dev/text/template
    // the main variable you can access is `{{.Args}}` which is a map of key=>value.
    // here we cat the data from hello.txt which we mounted before, then echo the value from
    // the arguments map exists in a key called message (just like $args['message']).
    command = "cat ./hello.txt && echo {{.Args.message}}"

    // after how long time should we kill the job?
    // a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, 
    // such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
    // empty means "no time limit".
    max_execution_time = ""

    // sometimes when you run this service within a cloud engine like k8s or docker container
    // you will have to define multiple configmaps/volumes mounts,
    // to simplify this job, we can only mount exeq configurations file and add any other file required by the job
    // into the following mounts block.
    mount "./hello.txt" {
        content = "hello world"
    }
}

Exeq Binary

exeq is an intuitive cli app, you can just write exeq help from your shell and go with its help. the binary consists of the following subcommands

   queue:work     start the queue worker(s)
   enqueue:macro  submit macro to the queue
   enqueue:cmd    submit a raw shell command to the queue
   queue:jobs     list the jobs
   queue:stats    show queue stats
   serve:http     start the http server which enables you to enqueue macros and inspect the queue via /metrics endpoint with the help of promethues
   help, h        Shows a list of commands or help for one command

Steps

  • Run a queue daemon via exeq queue:work
  • Optional run the http server as per your needs.
  • Start submitting commands either via:
    • HTTP API POST /enqueue/{MACRO_NAME}, this endpoint accepts a json message which will be passed to the underlying shell command as args,
    • CLI
      • exeq enqueue:cmd echo hello world
      • exeq enqueue:macro MACRO_NAME -a k=v -a k2=v2
  • You may want to see the queue stats via:
    • CLI:
      • exeq queue:stats
      • watch exeq queue:stats
    • HTTP API:
      • GET /
      • GET /metrics (a promethues metrics endpoint)
  • You may want to list jobs history exeq queue:jobs
Owner
Mohammed Al Ashaal
a software engineer 🤓
Mohammed Al Ashaal
Similar Resources

ap 是一个 shell 工具,可以让其它 shell 命令的输出能够自动进入交互翻页模式

ap -- auto-pager ap 是一个 shell 工具,可以让其它 shell 命令的输出能够自动进入交互翻页模式。 ap 由两部分组成,一个 Go 语言编写的二进制程序,负责捕获命令的输出并支持翻页, 和一组 shell 脚本,负责为用户指定的命令清单创建与之同名的 wrapper。 经

Apr 12, 2022

🧑‍💻📊 Show off your most used shell commands

🧑‍💻📊 Show off your most used shell commands

tsukae 🧑‍💻 📊 Tsukae, 使え - means use in Japanese (so it refers to commands that you use) Built on top of termui and cobra Big shoutout to jokerj40 f

Dec 17, 2022

Tool for shell commands execution, visualization and alerting. Configured with a simple YAML file.

Tool for shell commands execution, visualization and alerting. Configured with a simple YAML file.

Sampler. Visualization for any shell command. Sampler is a tool for shell commands execution, visualization and alerting. Configured with a simple YAM

Dec 28, 2022

Create new commands from your shell history or terminal.

overdub Create new commands from your shell history or terminal. TODO list for initial release Filter out unlikely commands (e.g. package managers) fr

Aug 9, 2022

Testing local and remote shell commands in Go

Testing local and remote shell commands in Go. This is an (intentionally simplified) example of how unix shell commands can be unit-tested in Go. The

Nov 30, 2021

This utility verifies all commands used by a shell script against an allow list

Find external commands required by shell scripts When writing shell scripts that need to run portably across multiple hosts and platforms, it's useful

Aug 15, 2022

A Go library and common interface for running local and remote commands

go-runcmd go-runcmd is a Go library and common interface for running local and remote commands providing the Runner interface which helps to abstract

Nov 25, 2021

Command Line Interface for Terraform Enterprise/Cloud ( tecli )

Command Line Interface for Terraform Enterprise/Cloud ( tecli )

In a world where everything is Terraform, teams use Terraform Cloud API to manage their workloads. TECLI increases teams productivity by facilitating such interaction and by providing easy commands that can be executed on a terminal or on CI/CD systems.

Dec 16, 2022

A multiple reverse shell sessions/clients manager via terminal written in go

A multiple reverse shell sessions/clients manager via terminal written in go

Nov 9, 2022
GodSpeed is a robust and intuitive manager for reverse shells.
GodSpeed is a robust and intuitive manager for reverse shells.

GodSpeed is a robust and intuitive manager for reverse shells. It supports tab-completion, verbose listing of connected hosts and easy interaction with selected shells by passing their corresponding ID.

Dec 12, 2022
Go package for running Linux distributed shell commands via SSH.
Go package for running Linux distributed shell commands via SSH.

Go package for running Linux distributed shell commands via SSH.

Dec 7, 2022
Go Library to Execute Commands Over SSH at Scale
Go Library to Execute Commands Over SSH at Scale

Go library to handle tens of thousands SSH connections and execute the command(s) with higher-level API for building network device / server automation.

Dec 9, 2022
GC2 is a Command and Control application that allows an attacker to execute commands on the target machine using Google Sheet and exfiltrate data using Google Drive.
GC2 is a Command and Control application that allows an attacker to execute commands on the target machine using Google Sheet and exfiltrate data using Google Drive.

GC2 GC2 (Google Command and Control) is a Command and Control application that allows an attacker to execute commands on the target machine using Goog

Dec 13, 2022
Slack remote terminal - execute commands on remote host using slack slash command

slackRT Slack remote terminal - execute commands on remote host using slack slash command Installation Go to api.slack.com/apps and sign in and create

Jul 12, 2022
Envp - ENVP is cli wrapper that sets environment variables by profile when you execute the command line

ENVP ENVP is cli wrapper that sets environment variables by profile based config

Nov 7, 2022
A modern and intuitive terminal-based text editor
A modern and intuitive terminal-based text editor

micro is a terminal-based text editor that aims to be easy to use and intuitive, while also taking advantage of the capabilities of modern terminals

Jan 7, 2023
The Todo List / Task Manager for Geeks in command line
The Todo List / Task Manager for Geeks in command line

The CLI To-Do List / Task Manager for Geeks ??‍?? Developer / DevOps / Sysadmin? A command line hero? ?? Live with the dark terminal? ?? Think in Mark

Dec 15, 2022
Grit is an experimental personal task manager that represents tasks as nodes of a multitree
Grit is an experimental personal task manager that represents tasks as nodes of a multitree

Grit is an experimental personal task manager that represents tasks as nodes of a multitree, a class of directed acyclic graphs. The structure en

Jan 2, 2023
cTRL is a server for remote execution of pending tasks and commands in real time, supporting a queue with continuous thread limiting and throttling.

Документация на русском: https://github.com/eltaline/ctrl/blob/master/README-RUS.md cTRL is a server written in Go language that uses a modified versi

Mar 3, 2022