Persistent queue in Go based on BBolt

Persistent queue

Persistent queue based on bbolt DB.

Supposed to be used as embeddable persistent queue to any Go application.

Features:

  • messages are not limited by RAM size, stream-oriented data
  • works good for small (automatic inlining) and for big streams
  • simple, portable storage structure
  • go-routine safe
  • supports multiple-writers and multiple-readers
  • supports ack/nack (commit with discard)

See go-docs for examples and details.

Requirements:

  • go 1.17

Motivation

I wanted to create an application for resource-constrained devices (ie: AWS Lightsail, Raspberry Pi Zero W, etc..) which should store (always) and forward (eventually) information with very unreliable network connection (ie: days without a link). Information could be small (sensors) or huge (webhooks). Devices themselves may experience a power outage.

It means:

  • stored information (number and individual records) may grow much above RAM
  • stored information should not be marked as processed before explicit commit
  • full-packed solutions like RabbitMQ/Kafka/etc can not be applied
  • due to distributed nature of the system duplicates inevitable, however, should be a cheap way to deduplicate (ie: unique ID)
Owner
Aleksandr Baryshnikov
Aleksandr Baryshnikov
https://github.com/reddec/pqueue
Similar Resources

Queue with NATS Jetstream to remove all the erlangs from cloud

Saf in Persian means Queue. One of the problems, that we face on projects with queues is deploying RabbitMQ on the cloud which brings us many challenges for CPU load, etc. I want to see how NATS with Jetstream can work as the queue to replace RabbitMQ.

Dec 15, 2022

A fast durable queue for Go

pqueue - a fast durable queue for Go pqueue is thread-safety, serves environments where more durability is required (e.g., outages last longer than me

Oct 16, 2022

Redis as backend for Queue Package

Redis as backend for Queue Package

redis Redis as backend for Queue package Setup start the redis server redis-server start the redis cluster, see the config # server 01 mkdir server01

Oct 16, 2022

NSQ as backend for Queue Package

NSQ as backend for Queue Package

NSQ as backend for Queue Package

Jul 4, 2022

Kudruk helps you to create queue channels and manage them gracefully.

kudruk Channels are widely used as queues. kudruk (means queue in Turkish) helps you to easily create queue with channel and manage the data in the qu

Feb 21, 2022

Chanman helps you to create queue channels and manage them gracefully.

chanman Channels are widely used as queues. chanman (Channel Manager) helps you to easily create queue with channel and manage the data in the queue.

Oct 16, 2021

KubeMQ is a Kubernetes native message queue broker

KubeMQ Community is the open-source version of KubeMQ, the Kubernetes native message broker. More about KubeMQ

Nov 20, 2021

Golang Delay Queue

[gdq] Golang Delay Queue GDQ is a library that leverage db or cache to be setup as a delay queue. For current version, Only redis can adapt to this li

Jan 15, 2022

A lightweight, thread-safe FIFO queue with fixed capacity.

FIFO Queue Thread-safe, lightweight, tested FIFO queue with fixed size, which is built upon list for performance ;-) This implement is inspired by htt

Dec 16, 2021
Related tags
Messaging pqueue
Machinery is an asynchronous task queue/job queue based on distributed message passing.
 Machinery is an asynchronous task queue/job queue based on distributed message passing.

Machinery Machinery is an asynchronous task queue/job queue based on distributed message passing. V2 Experiment First Steps Configuration Lock Broker

Jan 7, 2023
A Multi Consumer per Message Queue with persistence and Queue Stages.
 A Multi Consumer per Message Queue with persistence and Queue Stages.

CrimsonQ A Multi Consumer per Message Queue with persistence and Queue Stages. Under Active Development Crimson Queue allows you to have multiple cons

Jul 30, 2022
Embedded, Fast and Persistent bigqueue implementation

bigqueue bigqueue provides embedded, fast and persistent queue written in pure Go using memory mapped (mmap) files. bigqueue is now thread safe as wel

Sep 24, 2022
A lightweight, distributed and reliable message queue based on Redis

nmq A lightweight, distributed and reliable message queue based on Redis Get Started Download go get github.com/inuggets/nmq Usage import "github.com

Nov 22, 2021
Asynq: simple, reliable, and efficient distributed task queue in Go
 Asynq: simple, reliable, and efficient distributed task queue in Go

Asynq Overview Asynq is a Go library for queueing tasks and processing them asynchronously with workers. It's backed by Redis and is designed to be sc

Dec 30, 2022
RapidMQ is a pure, extremely productive, lightweight and reliable library for managing of the local messages queue

RapidMQ RapidMQ is a pure, extremely productive, lightweight and reliable library for managing of the local messages queue in the Go programming langu

Sep 27, 2022
redisqueue provides a producer and consumer of a queue that uses Redis streams

redisqueue redisqueue provides a producer and consumer of a queue that uses Redis streams. Features A Producer struct to make enqueuing messages easy.

Dec 29, 2022
A single binary, simple, message queue.

MiniQueue A stupid simple, single binary message queue using HTTP/2. Most messaging workloads don't require enormous amounts of data, endless features

Nov 9, 2022
dque is a fast, embedded, durable queue for Go

dque - a fast embedded durable queue for Go dque is: persistent -- survives program restarts scalable -- not limited by your RAM, but by your disk spa

Jan 8, 2023
Gue is Golang queue on top of PostgreSQL that uses transaction-level locks.

Gue is Golang queue on top of PostgreSQL that uses transaction-level locks.

Jan 4, 2023