Processing Nomad Events Stream

Nomad Events Sink

Nomad Events Sink is an events collection agent which uses Nomad Events SDK to fetch events.

Events can help debug the cluster state and alert operators about new deployments, failing allocations, node updates etc. Nomad emits these events in a channel and provides an SDK to access these events.

The missing piece was a tool to ingest these events and analyze/visualize them later. This is where nomad-events-sink agent comes into picture and helps to ingest these events in multiple configurable Sinks.

The tool is designed to be generic in order to handle many use-cases of processing events. Multiple Sink Providers can be created and events will be handled by them. Common usecases included storing events for long term use (Vector->Loki), alerting on cluster state changes (Slack/Rocketchat webhooks) etc.

How does it work?

  • A subscription for each topic is created and new events are watched.
  • Whenever a new event comes, it gets added to a Sink channel.
  • Multiple background workers (Goroutines) are listening to this channel, processing events in FIFO.
  • These workers batch events and push to upstream providers (for eg HTTP/S3 etc).
  • A background worker commits the event index state to a file on disk. This is done so that the program is able to pickup events from where it left before shutting down.

Batching Events

This program uses 2 batch strategies in order to avoid an inefficient process of making a request for every incoming event:

  • idle_timeout: If a batch is in memory for more than idle_timeout duration, it is flushed to providers.
  • events_count: If a batch has more events than events_counts, it is flushed to providers.

Deploy

You can choose one of the various deployment options:

Nomad

To deploy on a Nomad cluster, you can refer to this jobspec.

Binary

Grab the latest release from Releases.

To run:

$ ./nomad-events-sink.bin --config config.toml

Docker

docker pull ghcr.io/mr-karan/nomad-events-sink:latest

Quick Start

Refer to this example for quickly getting started with ingesting Deployment Events to Loki using Vector.

Configuration

Refer to config.sample.toml for a list of configurable values.

Environment Variables

All config variables can also be populated as env vairables by prefixing NOMAD_EVENTS_SINK_ and replacing . with __.

For eg: app.data_dir becomes NOMAD_EVENTS_SINK_app__data_dir.

Nomad API client reads the following environment variables:

  • NOMAD_TOKEN
  • NOMAD_ADDR
  • NOMAD_REGION
  • NOMAD_NAMESPACE
  • NOMAD_HTTP_AUTH
  • NOMAD_CACERT
  • NOMAD_CAPATH
  • NOMAD_CLIENT_CERT
  • NOMAD_CLIENT_KEY
  • NOMAD_TLS_SERVER_NAME
  • NOMAD_SKIP_VERIFY

You can read about them in detail here.

Sink Providers

  • Currently only HTTP Provider is implemented. There's a full working example of ingesting events to the HTTP server provided by Vector and dumping them to Loki.

Contribution

Please feel free to open a new issue for bugs, new features (like Sink Providers), feedback etc.

LICENSE

LICENSE

Owner
Karan Sharma
Tech @zerodha Interested in Observability, Containers and Distributed Systems.
Karan Sharma
Similar Resources

Nomad-driver-await-dependency - A Nomad driver that acts as blocker for subsequent task until a given Consul service has reached a given state

Nomad Skeleton Driver Plugin Skeleton project for Nomad task driver plugins. Thi

Feb 12, 2022

This POC is built with the goal to collect events/logs from the host systems such as Kubernetes, Docker, VMs, etc. A buffering layer is added to buffer events from the collector

This POC is built with the goal to collect events/logs from the host systems such as Kubernetes, Docker, VMs, etc. A buffering layer is added to buffer events from the collector

What is does This POC is build with the goal to collect events/logs from the host systems such as Kubernetes, docker, VMs etc. A buffering layer is ad

Nov 11, 2022

Stream database events from PostgreSQL to Kafka

PSQL-Streamer This service receives the database events from PostgreSQL using logical replication protocol and feeds them to sinks based on the config

Dec 20, 2022

Go Stream, like Java 8 Stream.

Go Stream, like Java 8 Stream.

Dec 1, 2022

Go Collection Stream API, inspired in Java 8 Stream.

GoStream gostream 是一个数据流式处理库。它可以声明式地对数据进行转换、过滤、排序、分组、收集,而无需关心操作细节。 Changelog 2021-11-18 add ToSet() collector Roadmap 移除go-linq依赖 Get GoStream go get

Nov 21, 2022

A stream processing API for Go (alpha)

A stream processing API for Go (alpha)

A data stream processing API for Go (alpha) Automi is an API for processing streams of data using idiomatic Go. Using Automi, programs can process str

Dec 28, 2022

A lightweight stream processing library for Go

A lightweight stream processing library for Go

go-streams A lightweight stream processing library for Go. go-streams provides a simple and concise DSL to build data pipelines. Wiki In computing, a

Dec 31, 2022

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

Overview Gazette makes it easy to build platforms that flexibly mix SQL, batch, and millisecond-latency streaming processing paradigms. It enables tea

Dec 12, 2022

💥 Fusion is a tiny stream processing library written in Go.

💥 Fusion Fusion is a tiny stream processing library written in Go. See reactor for a stream processing tool built using fusion. Features Simple & lig

Jun 30, 2021

Stream processing stuff for Go

GoStream Type safe Stream processing library inspired in the Java Streams API. Table of contents Requirements Usage examples Limitations Performance C

Dec 26, 2022

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

Jan 5, 2023

DigitalOcean Droplets target plugin for HashiCorp Nomad Autoscaler

Nomad DigitalOcean Droplets Autoscaler The do-droplets target plugin allows for the scaling of the Nomad cluster clients via creating and destroying D

Dec 8, 2022

A terminal UI (TUI) for HashiCorp Nomad

A terminal UI (TUI) for HashiCorp Nomad

Damon - A terminal Dashboard for HashiCorp Nomad Damon is a terminal user interface (TUI) for Nomad. It provides functionality to observe and interact

Jan 6, 2023

OpenAPI specification and related artifacts for HashiCorp Nomad

Overview This repository contains the HashiCorp Nomad OpenAPI specification and related artifacts. The OpenAPI specification defines a machine-readabl

Dec 14, 2022

A cron-like strategy plugin for HashiCorp Nomad Autoscaler

Nomad Autoscaler Cron Strategy A cron-like strategy plugin, where task groups are scaled based on a predefined scheduled. job "webapp" { ... group

Feb 14, 2022

The Operator Pattern, in Nomad

Nomad Operator Example Repostiory to go along with my The Operator Pattern in Nomad blog post. Usage If you have tmux installed, you can run start.sh

May 12, 2022

Watch Nomad allocations and update logshipper config

Nomad Logger This is a simple Go application that polls the Nomad API for all al

Apr 9, 2022

Nomad plugin for reserving device mappings used by ebs devices.

Nomad Skeleton Device Plugin Skeleton project for Nomad device plugins. This project is intended for bootstrapping development of a new device plugin.

Jan 5, 2022

Hetzner-dns-updater - A simple tool to update a DNS record via Hetzner DNS API. Used for simple HA together with Nomad

hetzner-dns-updater A small utility tool to update a single record via Hetzner D

Feb 12, 2022
Comments
  • feat: make stream a public package

    feat: make stream a public package

    This commit makes stream as a resuable package. This allows people to write arbitary programs utilising Nomad events stream and not just limited to sink providers.

Related tags
Nomad Coin - Nomad Coin study

nomad_Coin nomad_Coin study 노마드코인 스터디 Rough 상수 및 변수 ** 상수선언 const name string =

Feb 2, 2022
💥 Fusion is a tiny stream processing library written in Go.

?? Fusion Fusion is a tiny stream processing library written in Go. See reactor for a stream processing tool built using fusion. Features Simple & lig

Jun 30, 2021
Exfiltrates join/part events from a dedicated Valheim log and transmits them to a Herald webhook.

Exfiltrates join/part events from a dedicated Valheim log and transmits them to a Herald webhook.

Feb 6, 2022
Monitor usb hotplug events (Linux)

USBMon Thin udev wrapper to simplify usb device add/remove monitoring. // monitor USB hotplug events package main import ( "context" "fmt" "githu

Aug 1, 2022
Measurements stream joiner.

sensors Measurements stream joiner

Dec 13, 2021
A Go package to allow you to read and write from the serial port as a stream of bytes.

Serial A Go package to allow you to read and write from the serial port as a stream of bytes. Details It aims to have the same API on all platforms, i

Jan 6, 2023
Exercise for solve problem data processing, performance and something wrong in passing data

Citcall Exercise Exercise for solve problem data processing, performance and something wrong in passing data Pengolahan data data processing - Readme

Nov 25, 2021
Tidb - An open-source NewSQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads
Tidb - An open-source NewSQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads

What is TiDB? TiDB ("Ti" stands for Titanium) is an open-source NewSQL database

Jan 5, 2022
Nomad Pack is a templating and packaging tool used with HashiCorp Nomad.

Nomad Pack is a templating and packaging tool used with HashiCorp Nomad.

Jan 4, 2023
Nomad Coin - Nomad Coin study

nomad_Coin nomad_Coin study 노마드코인 스터디 Rough 상수 및 변수 ** 상수선언 const name string =

Feb 2, 2022