Super fault-tolerant gateway for HTTP clusters, written in Go. White paper for reference - https://github.com/gptankit/serviceq-paper

ServiceQ Build Status GoDoc

ServiceQ is a fault-tolerant gateway for HTTP clusters. It employs probabilistic routing to distribute load during partial cluster shutdown (k/n nodes experiencing downtimes, timeouts, connection loss etc) and buffers requests during total cluster shutdown (n nodes down). The buffered requests are forwarded in FIFO order when the cluster is available next.

Below graph shows the routing probability (P) on a down node (D) in a 8-node cluster with respect to number of requests (r). Notice how quickly the routing probability on D reduces as the requests on D start to fail. Depending on the rate of request, it will only take a few seconds (sometime even milliseconds) to move all requests away from D, thus ensuring more requests are routed to healthier nodes.

Note that, even when requests keep failing on D (however less), ServiceQ retries them on other nodes until they succeed. If they do not succeed on any of the nodes, they are buffered and periodically retried on the cluster (using the same approach above), until they succeed.

Noticeable features

  • HTTP Load Balancing
  • Probabilistic node selection based on error feedback
  • Failed request buffering and deferred forwarding
  • Upfront request buffering
  • Request retries
  • Concurrent connections limit
  • Complete TLS/SSL support (automatic and manual)

Here are the steps to run ServiceQ -

Download

Clone the project into any directory in your workspace (say 'serviceq')

$ mkdir serviceq
$ git clone https://github.com/gptankit/serviceq serviceq/

Change into directory serviceq

How to Build

$ make ('make build' will also work)

Optional: make with debug symbols removed (~25% size reduction)

$ make build-nodbg

This will create a Go binary serviceq in the current directory

How to Install

Make sure the current user has root privileges, then -

$ make install

This will create a folder serviceq in /usr/local directory and copy the serviceq binary to /usr/local/serviceq and sq.properties file (serviceq configuration) to /usr/local/serviceq/config.

How to Run

Before running, make sure the mandatory configurations in /usr/local/serviceq/config/sq.properties are set (LISTENER_PORT, PROTO, ENDPOINTS, CONCURRENCY_PEAK) -

#sq.properties

#Port on which serviceq listens on
LISTENER_PORT=5252

#Protocol the endpoints listens on -- 'http' for both http/https
PROTO=http

#Endpoints seperated by comma (,) -- no spaces allowed, can be a combination of http/https
ENDPOINTS=http://my.server1.com:8080,http://my.server2.com:8080,http://my.server3.com:8080

#Concurrency peak defines how many max concurrent connections are allowed to the cluster
CONCURRENCY_PEAK=2048

Also, verify timeout value (default is set to 5s). Low value is preferable as it allows retries to be faster -

#Timeout (s) is added to each outgoing request to endpoints, the existing timeouts are overriden, value of -1 means no timeout
OUTGOING_REQUEST_TIMEOUT=5

By default deferred queue is enabled with all methods and routes allowed. These options can be controlled as -

#Enable deferred queue for requests on final failures (cluster down)
ENABLE_DEFERRED_Q=true

#Request format allows given method/route on deferred queue -- picked up if ENABLE_DEFERRED_Q is true
#Q_REQUEST_FORMATS=POST /orders,PUT,PATCH,DELETE
#Q_REQUEST_FORMATS=ALL
Q_REQUEST_FORMATS=POST,PUT,PATCH,DELETE

(Note that Q_REQUEST_FORMATS is also considered if ENABLE_UPFRONT_Q is true)

After all is set -

$ sudo /usr/local/serviceq/serviceq

Refer wiki for more details: https://github.com/gptankit/serviceq/wiki

Feel free to play around and post feedbacks

Owner
Ankit Gupta
Engineering @walmartlabs, IIIT-Hyderabad CS
Ankit Gupta
Similar Resources

Reference implementation of the PLAN Data Model and core components

PLAN is a free and open platform for groups to securely communicate, collaborate, and coordinate projects and activities.

Nov 2, 2021

Reference Golang implementation of the Alphanet - Network of Momentum Phase 0

Zenon Node Reference Golang implementation of the Alphanet - Network of Momentum Phase 0. Building from source Building znnd requires both a Go (versi

Dec 25, 2022

Kubernetes Custom Resource API Reference Docs generator

Kubernetes Custom Resource API Reference Docs generator If you have a project that is Custom Resource Definitions and wanted to generate API Reference

Dec 7, 2021

whatsup is the reference server implementation for the fmrl protocol

whatsup whatsup is the reference server implementation for the fmrl protocol. Currently whatsup has no web interface, but may gain one in the future.

Mar 25, 2022

Golang Super Simple Load Balance

SSLB (Super Simple Load Balancer) ver 0.1.0 It's a Super Simple Load Balancer, just a little project to achieve some kind of performance. Features Hig

Dec 18, 2022

Super simple tcp intranet penetration proxy program

A super easy to configure tcp intranet penetration proxy program that forwards intranet tcp ports to public network servers. Tested proxies for intranet HTTP services, windows remote desktop, ssh access and other scenarios.

Dec 11, 2022

HTTP, HTTP2, HTTPS, Websocket debugging proxy

HTTP, HTTP2, HTTPS, Websocket debugging proxy

English | 简体中文 We recommend updating whistle and Node to ensure that you receive important features, bugfixes and performance improvements. Some versi

Dec 31, 2022

Local Portable HTTP/HTTPS Proxy

SkelgoKey Portable Local Web Proxy - Creates a local proxy that bypasses any network certificate checks - USAGES Windows .\SkeletonKey.exe (web addres

Oct 13, 2021
Comments
  • Getting 503 Request Buffered

    Getting 503 Request Buffered

    With the sq.properties listed below, if I query once I get results. But if I put any kind of concurrent load through the serviceq I immediately get a 503 (Request Buffered). Not sure what I am doing wong.

    `#ServiceQ Configuration File

    #------------------#

    Cluster Settings

    #------------------#

    #Port on which serviceq listens on LISTENER_PORT=5252

    #Protocol the endpoints listens on -- 'http' for both http/https PROTO=http

    #Endpoints seperated by comma (,) -- no spaces allowed, can be a combination of http/https ENDPOINTS=https://c1.xxx.com,https://c2.xxx.com,https://c3.xxx.com

    #Concurrency peak defines how many max concurrent connections are allowed to the cluster of endpoints defined above CONCURRENCY_PEAK=8096

    #Timeout (ms) is added to each outgoing request to endpoints, the existing timeouts are overriden, value of -1 means no timeout OUTGOING_REQUEST_TIMEOUT=300000

    #Interval (ms) between two retries -- recommended 0 for best performance RETRY_GAP=0

    #-------------------------#

    Deferred Queue Settings

    #-------------------------#

    #Enable deferred queue for requests on final failures (cluster down) ENABLE_DEFERRED_Q=true

    #Request format enables deferred queue on only the mentioned method and route -- picked up if ENABLE_DEFERRED_Q is true #DEFERRED_Q_REQUEST_FORMATS=POST /orders,PUT,PATCH,DELETE DEFERRED_Q_REQUEST_FORMATS=ALL #DEFERRED_Q_REQUEST_FORMATS=POST,PUT,PATCH,DELETE

    #-------------------#

    Response Settings

    #-------------------#

    #Custom response headers to be forwarded to client -- no spaces allowed #CUSTOM_RESPONSE_HEADERS=Connection: keep-alive|Server CUSTOM_RESPONSE_HEADERS=Server

    #Keep Alive Timeout (ms), value of -1 means no timeout KEEP_ALIVE_TIMEOUT=120000

    #--------------#

    SSL Settings

    #--------------#

    #Use SSL connection SSL_ENABLE=false

    #Path to certificate and private keys (do not provide quotes "") -- picked up if SSL_ENABLE is true SSL_CERTIFICATE_FILE=/usr/certs/cert.pem SSL_PRIVATE_KEY_FILE=/usr/certs/key.pem `

  • Suggestion/Question: deferred-all

    Suggestion/Question: deferred-all

    I have a use case where I would like to defer ALL requests, no matter the outcome. What I need is to respond to the original request as fast as possible with some pre-defined response, and then let ServiceQ handle a slower back-end. Obviously, the original request would always succeed and there would be no direct way to tell if the deferred request succeeded or not. I checked the documentation and learned about the deferred queue, but from what I understand it would only kick-in when a request fail, and not for all cases. Is this something that could be implemented easily? Would it be useful to others?

  • Observing Queues

    Observing Queues

    Hi,

    I'm interested in using this library as a reverse proxy+queuing solution to protect services that get periodically hammered with requests. I really like the upfront queue feature, however I'm wondering about the easiest way to gain some observability into queue status ie. number of requests in the queue, etc.

    Any thoughts on how best to observe?

  • Ask: Add persistent queue for deferred requests

    Ask: Add persistent queue for deferred requests

    The current deferred queue implementation uses Go buffered channels which are inherently in-memory structures. As such, they lose all buffered requests in case of service/node restarts and are also limited by system's memory. Few users have requested to add a persistent queue as well. Although there are low chances of queue filling up unless there is a problem elsewhere in the cluster, but I do see certain benefits of building a persistent deferred queue as an alternative -

    • Not limited by memory
    • Preserve requests across restarts
    • Ordering of requests across serviceq nodes
The Durudex gateway combines all durudex services so that it can be used through a single gateway.

The Durudex gateway combines all durudex services so that it can be used through a single gateway.

Dec 13, 2022
Grpc-gateway-map-null - gRPC Gateway test using nullable values in map

Demonstrate gRPC gateway behavior with nullable values in maps Using grpc-gatewa

Jan 6, 2022
Openldap (LDAP) binding for Golang (go) ; no more support ; you may have a look at https://github.com/go-ldap/ldap

OpenLDAP this is Openldap binding in GO language. I don't work any more with golang, so, please fork this project. Installation : Installation is easy

Mar 4, 2021
An Etsy StatsD (https://github.com/etsy/statsd) implementation in Go

STATSD-GO Port of Etsy's statsd, written in Go. This was forked from https://github.com/amir/gographite to provide Ganglia submission support. USAGE U

Mar 5, 2021
Native ZooKeeper client for Go. This project is no longer maintained. Please use https://github.com/go-zookeeper/zk instead.

Native Go Zookeeper Client Library License 3-clause BSD. See LICENSE file. This Repository is No Longer Maintained Please use https://github.com/go-zo

Dec 19, 2022
Educational gRPC-based Bitcoin paper-trading server

gRPCoin: Programmatic paper trading platform using gRPC and Protobuf This is an educational project that allows you to trade cryptocurrencies competit

Dec 10, 2022
Certified paper with golang

Certified paper with golang

Nov 14, 2021
Drop-in replacement for Go net/http when running in AWS Lambda & API Gateway
Drop-in replacement for Go net/http when running in AWS Lambda & API Gateway

Package gateway provides a drop-in replacement for net/http's ListenAndServe for use in AWS Lambda & API Gateway, simply swap it out for gateway.Liste

Nov 24, 2022
No-frills all-in-one HTTP API gateway

SX: a minimal, declarative API gateway WARNING: not production ready. Use at your own risk! Need something better? Check out nginx, Caddy or Envoy SX

Dec 18, 2021
Exposure Notification Reference Server | Covid-19 Exposure Notifications

Exposure Notification Reference Key Server COVID‑19 Exposure Notifications API Exposure Notification Reference Key Server Documentation In our continu

Jan 7, 2023