HARAQA - High Availability Routing And Queueing Application

haraqa is designed to be a developer friendly, scalable message queue for data persistence and real-time data streaming between microservices. Haraqa provides high-throughput, low-latency, fault-tolerant pipelines for architectures of any size.

Table of Contents

• About the Project
  • Overview
  • Persistence and Replication
• Getting Started
  • API Docs
  • Server
  • Client
• Contributing
• License

About the Project

Overview

Haraqa is meant for handling and persisting data in a distributed system. One or more servers can be used to send and receive messages. Each server has a set of 'topics', a set of messages stored in the order received.

A Haraqa client can produce and/or consume from a server's topics. Clients can produce messages one at a time or in batches. Clients consume messages by making a request for a specific offset and limit. The messages can be consumed one at a time or in batches.

Persistence and Replication

Each server, after receiving a message from a producer, can save the message to multiple volumes. These volumes are meant to be distributed in the architecture, such as having multiple PersistentVolumes in a Kubernetes cluster, EBS in AWS, or Persistent Disks in Google Cloud. The server reads messages from the last volume when sending to consumer clients.

If a volume is removed or corrupted during a restart the server repopulates the data from the other volumes.

Getting started

API Docs

• Swagger API Documentation
• Redocs API Documentation
• Swagger yaml

The docker server also includes local api documentation at the /docs/redocs and /docs/swagger endpoints.

Server

The recommended deployment strategy is to use Docker

docker run -it -p 4353:4353 -v $PWD/vol1:/vol1 haraqa/haraqa /vol1

docker run -it [port mapping] [volume mounts] haraqa/haraqa [flags] [volumes]

To run from source, navigate to cmd/server and run the main.go file.

cd cmd/server
go run main.go vol1

Flags:

  -http    uint    Port to listen on (default 4353)
  -cache   boolean Enable queue file caching (default true)
  -cors    boolean Enable CORS (default true)
  -docs    boolean Enable Docs pages (default true)
  -entries integer The number of msg entries per queue file before creating a new file (default 5000)
  -limit   integer Default batch limit for consumers (default -1)
  -ballast integer Garbage collection memory ballast size in bytes (default 1073741824)
  -prometheus boolean Enable prometheus metrics (default true)

Volumes:

Volumes will be written to in the order given and recovered from in the reverse order. Consumer requests are read from the last volume. For this reason it's recommended to use a local volume last.

For instance, given

docker run haraqa/haraqa /vol1 /vol2 /vol3

When a client produces a message, it will be written to /vol1, then /vol2, then /vol3. When a client consumes a message, it will be read from /vol3.

During recovery, if data exists in /vol3 it will be replicated to volumes /vol1 and /vol2. If /vol3 is empty, /vol2 will be replicated to /vol1 and /vol3.

Client

go get github.com/haraqa/haraqa

Contributing

We want this project to be the best it can be and all feedback, feature requests or pull requests are welcome.

License

MIT © 2019 haraqa and contributors. See LICENSE for more information.