A simple queueing system for long-running commands

qme (queue me)

A simple queueing system for long-running commands. It allows you to queue up shell commands from anywhere, and run them in order.

This is useful for enqueueing long-running commands sharing a limited resource, like a video encoding (which maxes out CPU), rsync'ing files (which take up all upload bandwidth), or running a build script (which takes up all CPU).

If the program you're running has no built-in queueing functionality, and you have no pre-determined list of jobs, such that running a shell script isn't an option, this is a simple way to get it done.

Usage

In any terminal simply prefix your command with qme

sleep 5  # executes the command directly
./qme sleep 5  # queues and executes the command

This will enqueue the command and start executing it right away, piping its stdout and stderr to the terminal, but it will also keep an RPC server running in the background.

$ ./qme sleep 5
2022/01/21 10:54:12 enqueueing 'sleep'
2022/01/21 10:54:12 assuming server role
2022/01/21 10:54:12 listening on /tmp/qme.sock
2022/01/21 10:54:12 started executing 'sleep' with pid 62775
2022/01/21 10:28:30 finished: exit status 0
2022/01/21 10:28:30 idling...
2022/01/21 10:28:50 idle timeout reached, shutting down

So when you enqueue another task before the server process shuts down (it timeouts in 20s), it will connect & enqueue the command on the server process, and it will be executed there.

# this will be executed on the server process
$ ./qme sleep 1
2022/01/21 10:54:13 connected. assuming client role
2022/01/21 10:54:13 enqueued 'sleep'

$ ./qme sleep 2
2022/01/21 10:54:14 connected. assuming client role
2022/01/21 10:54:14 enqueued 'sleep'
# server process accepts the command, and starts executing it  
...
2022/01/21 10:30:36 idling...
2022/01/21 10:54:13 enqueueing 'sleep'  # <-- command accepted into the queue
2022/01/21 10:54:14 enqueueing 'sleep'  # <-- 
2022/01/21 10:54:22 finished: exit status 0
2022/01/21 10:54:22 idling...
2022/01/21 10:54:22 started executing 'sleep' with pid 62804 # <-- command is now executing
2022/01/21 10:54:23 finished: exit status 0
2022/01/21 10:54:23 idling...
2022/01/21 10:54:23 started executing 'sleep' with pid 62805
2022/01/21 10:54:25 finished: exit status 0
2022/01/21 10:54:25 idling...
2022/01/21 10:54:45 idle timeout reached, shutting down # <-- server process shuts down if there's nothing to do

If the server already shut down, it will assume the server role and start executing & listening again. So no matter when & where you run qme, it will run the command now or after other queued commands finishes executing, but not at the same time.

TODO

  • Add tests
  • Respect os signals
  • Make idle timeout configurable
  • Support separate queues (e.g. one queue for CPU-heavy, another one for network-heavy, etc.)
  • Support command weights, so that important commands are executed first
Owner
Abdussamet Koçak
Abdussamet Koçak
Similar Resources

A lightweight event collection system.

A lightweight event collection system.

Honeypot A self-contained, multi-protocol streaming event collection system with ambitions to be as boring as benthos. Honeypot is primarily built for

Dec 6, 2022

Our notification system simplifies the process of sending notifications via email, SMS, and push notifications for multiple applications. It supports multiple providers, customizable templates, and is easy to integrate into any application.

Our notification system simplifies the process of sending notifications via email, SMS, and push notifications for multiple applications. It supports multiple providers, customizable templates, and is easy to integrate into any application.

Simplify Notification Management with Customizable Templates and Multi-Provider Integration ⭐️ Why Envoyer Nowadays, notifications play a crucial role

May 11, 2023

Asynq: simple, reliable, and efficient distributed task queue in Go

Asynq: simple, reliable, and efficient distributed task queue in Go

Asynq Overview Asynq is a Go library for queueing tasks and processing them asynchronously with workers. It's backed by Redis and is designed to be sc

Dec 30, 2022

Abstraction layer for simple rabbitMQ connection, messaging and administration

Abstraction layer for simple rabbitMQ connection, messaging and administration

Jazz Abstraction layer for quick and simple rabbitMQ connection, messaging and administration. Inspired by Jazz Jackrabbit and his eternal hatred towa

Dec 12, 2022

Go simple async message bus

Go simple async message bus

🚌 message-bus Go simple async message bus. 📖 ABOUT Contributors: Rafał Lorenz Want to contribute ? Feel free to send pull requests! Have problems, b

Dec 29, 2022

A simple pubsub package for go.

Package pubsub implements a simple multi-topic pub-sub library. Install pubsub with, go get github.com/cskr/pubsub This repository is a go module and

Dec 31, 2022

A single binary, simple, message queue.

MiniQueue A stupid simple, single binary message queue using HTTP/2. Most messaging workloads don't require enormous amounts of data, endless features

Nov 9, 2022

A dead simple Go library for sending notifications to various messaging services.

A dead simple Go library for sending notifications to various messaging services.

A dead simple Go library for sending notifications to various messaging services. About Notify arose from my own need for one of my api server running

Jan 7, 2023

Simple, high-performance event streaming broker

Styx Styx is a simple and high-performance event streaming broker. It aims to provide teams of all sizes with a simple to operate, disk-persisted publ

Nov 24, 2022
Comments
  • Bug: application doesn't terminate on receipt of signal interrupt ctrl+c

    Bug: application doesn't terminate on receipt of signal interrupt ctrl+c

    running as normal unprivileged user on linux. steps to reproduce:

    1. issue an invalid command e.g. qme --help
    2. observe that error message received and then application begins idling
    3. introduce signal interrupt key sequence ctrl+c
    4. application acknowledges interrupt signal received and states shutdown is beginning
    5. application hangs indefinitely and doesn't shut down
    2022/01/26 21:42:13 failed to execute command --help: exec: "--help": executable file not found in $PATH
    2022/01/26 21:42:13 idling...
    ^C2022/01/26 22:04:11 interrupt signal received
    2022/01/26 22:04:11 shutting down
    
  • Socket ought to be unique for the individual user

    Socket ought to be unique for the individual user

    As it is, qme listens on /tmp/qme.sock. This will fail in a multi-user environment where more that one user is attempting to queue jobs. Instead use a socket name based on e.g. the user's numerical id — i.e., /tmp/qme-1234.sock for the user with the numerical id 1234.

    Ideally, the socket should be created in the directory pointed to by the $TMPDIR variable, but a socket-per-user will do for a start.


    :link: https://en.wikipedia.org/wiki/TMPDIR

  • Please provide a simple help text

    Please provide a simple help text

    It would be helpful if the qme command accepted at least a -h option for a minimal help text — e.g.:

    $ qme -h
    usage: qme [-h] COMMAND [ARG ...]
    
    positional arguments:
      COMMAND     command to run
      ARG         optional argument(s)
    
    options:
      -h, --help  show this help message and exit
    

    As it is, qme will try ro queue -h as a command:

    $ qme -h
    2022/06/20 15:22:22 assuming server role
    2022/06/20 15:22:22 enqueueing '-h'
    2022/06/20 15:22:22 failed to execute command -h: exec: "-h": executable file not found in $PATH
    2022/06/20 15:22:22 idling...
    2022/06/20 15:22:22 listening on /tmp/qme.sock
    

    PS: A -V/--version option would also come in handy — e.g., for bug reports.

HARAQA - High Availability Routing And Queueing Application
HARAQA - High Availability Routing And Queueing Application

haraqa is designed to be a developer friendly, scalable message queue for data persistence and real-time data streaming between microservices. Haraqa provides high-throughput, low-latency, fault-tolerant pipelines for architectures of any size.

Nov 1, 2022
Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

Cadence Visit cadenceworkflow.io to learn about Cadence. This repo contains the source code of the Cadence server. To implement workflows, activities

Jan 9, 2023
Simple push notification system (android/fcm, apns/slideshow) written by Golang

Golang Push Notification Simple system push notification for android/fcm and apn

Dec 20, 2021
💨 A real time messaging system to build a scalable in-app notifications, multiplayer games, chat apps in web and mobile apps.
💨 A real time messaging system to build a scalable in-app notifications, multiplayer games, chat apps in web and mobile apps.

Beaver A Real Time Messaging Server. Beaver is a real-time messaging server. With beaver you can easily build scalable in-app notifications, realtime

Jan 1, 2023
Golang client for NATS, the cloud native messaging system.

NATS - Go Client A Go client for the NATS messaging system. Installation # Go client go get github.com/nats-io/nats.go/ # Server go get github.com/na

Jan 5, 2023
A generic oplog/replication system for microservices
A generic oplog/replication system for microservices

REST Operation Log OpLog is used as a real-time data synchronization layer between a producer and consumers. Basically, it's a generic database replic

Oct 23, 2022
Uniqush is a free and open source software system which provides a unified push service for server side notification to apps on mobile devices.

Homepage Download Blog/News @uniqush Introduction Uniqush (\ˈyü-nə-ku̇sh\ "uni" pronounced as in "unified", and "qush" pronounced as in "cushion") is

Jan 9, 2023
High-Performance server for NATS, the cloud native messaging system.
High-Performance server for NATS, the cloud native messaging system.

NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Fo

Jan 2, 2023
The Xiaomi message push service is a system-level channel on MIUI and is universal across the platform, which can provide developers with stable, reliable, and efficient push services.

Go-Push-API MiPush、JiPush、UMeng MiPush The Xiaomi message push service is a system-level channel on MIUI and is universal across the platform, which c

Oct 20, 2022
Go-notification - Realtime notification system with golang

Realtime notification system Used Apache kafka gRPC & PROTOBUF MongoDB restapi w

Aug 19, 2022