Script Based Alerting Manager

GitHub release (latest by date) Go Report Card Test codecov

logo.png

A Project in active development. Features may have breaking changes at any time before v1.0.0 version

Balerter is a scripts based alerting system.

In your script you may:

  • obtain needed data from different data sources (prometheus, clickhouse, postgres, external HTTP API etc.)
  • analyze data and make a decision about alert status
  • change Alerts statuses and receive notifications about it

In the example bellow we create one Clickhouse datasource, one scripts source and one alert channel. In the script we run query to clickhouse, check the value and fire the alert (or switch off it)

Notification channels

  • Slack
  • Telegram
  • Syslog
  • Desktop Notify
  • Email
  • Discord
  • Webhook
  • Prometheus Alertmanager
  • Prometheus AlertmanagerReceiver
  • Twilio Voice (phone calls)

Datasources

  • Clickhouse
  • Prometheus
  • Postgres
  • MySQL
  • Loki
  • Any external API with http lua module

Full documentation available on https://balerter.com

Example

docker pull balerter/balerter
docker run \
    -v /path/to/config.yml:/opt/config.yml \
    -v /path/to/scripts:/opt/scripts \ 
    -v /path/to/cert.crt:/home/user/db.crt \
    balerter/balerter -config=/opt/config.yml

Config file config.yml

scripts:
  folder:
    - name: debug-folder
      path: /opt/scripts
      mask: '*.lua'

datasources:
  clickhouse:
    - name: ch1
      host: localhost
      port: 6440
      username: default
      password: secret
      database: default
      sslMode: verified_full
      sslCertPath: /home/user/db.crt

channels:
  slack:
    - name: slack1
      url: https://hooks.slack.com/services/hash

Sample script rps.lua

-- @cron */10 * * * * *
-- @name script1

local minRequestsRPS = 100

local log = require("log")
local ch1 = require("datasource.clickhouse.ch1")

local res, err = ch1.query("SELECT sum(requests) AS rps FROM some_table WHERE date = now()")
if err ~= nil then
    log.error("clickhouse 'ch1' query error: " .. err)
    return
end

local resultRPS = res[1].rps

if resultRPS < minResultRPS then
    alert.error("rps-min-limit", "Requests RPS are very small: " .. tostring(resultRPS))
else
    alert.success("rps-min-limit", "Requests RPS ok")
end 

Also, you can to write tests!

An example:

-- @test script1
-- @name script1-test

test = require('test')

local resp = {
    {
        rps = 10
    }
} 

test.datasource('clickhouse.ch1').on('query', 'SELECT sum(requests) AS rps FROM some_table WHERE date = now()').response(resp)

test.alert().assertCalled('error', 'rps-min-limit', 'Requests RPS are very small: 10')
test.alert().assertNotCalled('success', 'rps-min-limit', 'Requests RPS ok')

See a documentation on https://balerter.com

Owner
Balerter
A Script Based Alerting Manager
Balerter
Comments
  • Implement support for Cadence

    Implement support for Cadence

    The built in cron support via https://github.com/robfig/cron is somewhat limiting in that it can only run on a single system and has no UI/tooling. Adding support for Cadence might not be that difficult and would allow for scaling balerter significantly.

    Example cron workflow

  • Add CSV en/decoding

    Add CSV en/decoding

    Currently there is json en/decoding. Mby it would be good to have a fast CSV en/decoder. Could be useful for anomaly detection later on...

    Mby https://github.com/FourierTransformer/ftcsv what do you think?

  • Discuss how to write docs

    Discuss how to write docs

    The current way of documentation is done via github pages which is quite inconvenient for new contributors.

    Maybe a better way would be to use something like https://readthedocs.org/ and add a link on the homepage? What do you think or do you have a suggestion for something else?

  • Inject time of previous job run into lua state

    Inject time of previous job run into lua state

    Inject the time of previous cron execution into context of script execution. This would allow for alerts to run queries between the execution time and the time of prior execution. Perhaps create a meta module to expose these?

    
    local meta = require("meta")
    log.info("time of prior execution: " .. meta.priorExecutionTime)
    

    Maybe there are some other useful bits of context to make available to scripts? Perhaps cron location?

  • Simple web ui for creating, editing lua script's

    Simple web ui for creating, editing lua script's

    Hi @negasus

    what do you think about a basic web ui for creating, editing lua scripts not global config. I think about something really simple like integrated webserver,basic auth,templates,html+bootstrap/mincss.

  • Не возможно использовать webhook chanel для алетов

    Не возможно использовать webhook chanel для алетов

    Попытался использовать webhook chanel для передачи алертов в свой кастомный апи для телеграмм ботов. curl --header 'Content-Type: application/json' --request 'POST' --data '{"chat_id":"-11111111","text":"Тестовая отправка сообщения"}' http://proxy-api-url:8080/sendMessage но в https://balerter.com/0.9.1/configuration/channels.html#type-webhook Я сделал пытался сделать по аналогии с curl

    channels:
      webhook:
        - name: wh1
          settings:
            url: http://proxy-api-url:8080/sendMessage
            method: POST
            payload:
              queryParams:
                chat_id: "-11111111"
                text: Алерт <----- Нельзя менять из скрипта
              body: body content <----- Нельзя менять из скрипта
            timeout: 5000
            headers:
                Content-Type: application/json
          ignore: false
    

    нельзя менять текст параметра text....

  • Implement support for parameters within messages

    Implement support for parameters within messages

    It would be quite helpful for a number of use cases if messages could contain an optional set of key value pairs. This would allow for scripts to extract values and dispatch them for later use. For example, webhooks could use these values in the body of the Post payload. Or slack/discord messages could embed them to add more context to alerts.

    Elastalert supports something similar to this whereby an alert can extract field values and then use them to add context to alerts.

    Example

    include:
      - match_body.host.hostname
      - match_body.user.name
      - match_body.source.ip
    
    alert_subject: "SSH abuse (repeat offender) on <{}> | <{}|Show Dashboard>"
    alert_subject_args:
      - match_body.host.hostname
      - kibana_link
    
  • Create some examples

    Create some examples

    Hi @negasus what do you think about adding a example folder with some more examples? Currently I play with following:

    Use yahoo finance API to fetch some predefined stocks and save the value into KV store. Next API call I check if the returned value is a predefined amount of % lower than the previous one and fire an alert.

  • Cron jobs

    Cron jobs

    #7 I removed @interval support (use @schedule instead), because cron package have the same functionality (@every <duration>). We have no stable version now, so I think it is a good idea.

  • /build/balerter/internal/runner/job.go:54 error run job

    /build/balerter/internal/runner/job.go:54 error run job

    2020-11-23T06:43:07.728Z        DEBUG   /build/balerter/internal/alert/provider/telegram/send.go:10     tg send message
    2020-11-23T06:43:07.728Z        ERROR   /build/balerter/internal/runner/job.go:54       error run job   {"script name": "disk-low-space-ubd", "error": "runtime error: invalid memory address or nil pointer dereference\nstack traceback:\n\t[G]: in function 'error'\n\t<string>:5: in main chunk\n\t[G]: ?"}
    github.com/balerter/balerter/internal/runner.(*Runner).runJob
            /build/balerter/internal/runner/job.go:54
    
  • sqllite storage not working

    sqllite storage not working

    When SQLite selected as storage for KV and alerts, Balerter falls with error:

    error create core storages manager, error create file storage, Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub
    

    Balerter version: v0.9.0

  • Implement support for elastic/opensearch

    Implement support for elastic/opensearch

    Adding support for Elastic / OpenSearch would open up a wide range of use cases for folks currently using elastalertor elastalert2.

    Work in progress

    I've started sketching support in this branch using OpenSearchand the Go client from @olivere. These were selected to avoid potential issues withElastic's viral licensing changes. Currently it supports connecting to a datastore and simple query_string based queries.

    There is a bit more work to do before it's ready for a PR, including fleshing out the tests, sorting out dependencies, and support for various auth mechanisms.

    Interested in feedback on general approach and how to make this best fit into the balerter design ethos.

    Startup up containers and add a document

    $ docker-compose up
    ....
    $  curl -H "Content-Type: application/json" -XPOST "http://localhost:9200/testindex/testtype" -d "{ \"foo\" : \"bar\"}"               [20:01:09]
    {"_index":"testindex","_type":"testtype","_id":"iM0nAXwBZaWvkaRHbMYd","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}%                              
    

    Define a simple script and run it

    -- @cron * * * * * *
    
    local log = require("log")
    local alert = require("alert")
    
    local es1 = require("datasource.elastic.es1")
    
    local res, err = es1.query("testindex", "foo: bar")
    if err ~= nil then
        log.error("elastic 'es1' query error: " .. err)
        return
    end
    
    if #res >0 then
      log.info("Got more than one hit")
    else
      log.info("Got no hits")
    end
    
    
    scripts {
      updateInterval = 5000
      folder "scripts"{
        path = "balerter-test/scripts"
        mask = "*.lua"
      }
    }
    
    datasources {
      elastic "es1" {
        host = "127.0.0.1"
        port = 9200
        scheme = "http"
        sniff = "false"
        timeout = "10000"
      }
    }
    
    
    channels {
      notify "notify1" {}
    }
    
    
    $ balerter -config balerter.hcl                                                                                                                                                         [20:02:32]
    {"level":"info","ts":1632106954.4304147,"msg":"balerter start","version":"undefined"}
    {"level":"info","ts":1632106954.4307108,"msg":"init scripts manager"}
    {"level":"info","ts":1632106954.4307454,"msg":"init datasources manager"}
    {"level":"info","ts":1632106954.442129,"msg":"init upload storages manager"}
    {"level":"info","ts":1632106954.442164,"msg":"init core storages manager"}
    {"level":"info","ts":1632106954.4421866,"msg":"init channels manager"}
    {"level":"info","ts":1632106954.4422402,"msg":"init runner"}
    {"level":"info","ts":1632106954.4422848,"msg":"run runner"}
    {"level":"info","ts":1632106955.067425,"msg":"Got more than one hit","scriptName":"folder.scripts.test"}
    ...
    ^C{"level":"info","ts":1632106956.6963909,"msg":"got os signal","signal":"interrupt"}
    {"level":"info","ts":1632106956.6964293,"msg":"stop jobs"}
    {"level":"info","ts":1632106956.6964521,"msg":"terminate"}
    
Simple unpacking script for Ezuri ELF Crypter
Simple unpacking script for Ezuri ELF Crypter

ezuri_unpack A simple unpacking script for the Ezuri ELF Crypter. Based on the analysis done by Ofer Caspi and Fernando Martinez of AT&T Alien Labs

Dec 15, 2022
A local snippet manager in Go

Snip A local snippet manager :D Snip provides simple CLI for saving and copying links for emojis, gifs or whatever you want. Requirements Install go i

May 24, 2021
A concurrent Download Manager written in Go

golang-download-manager A concurrent Download Manager written in Go Changes In main.go file paste the file url in fileUrl variable paste the path for

Aug 16, 2022
go generate based graphql server library
go generate based graphql server library

gqlgen What is gqlgen? gqlgen is a Go library for building GraphQL servers without any fuss. gqlgen is based on a Schema first approach — You get to D

Dec 29, 2022
A simple thread-safe, fixed size LRU written in Go. Based on dominictarr's Hashlru Algorithm. 🔃

go-hashlru A simple thread-safe, fixed size LRU written in Go. Based on dominictarr's Hashlru Algorithm. ?? Uses map[interface{}]interface{} to allow

Dec 5, 2022
A Golang tool to whitelist ASN's based on organization name

A Golang tool to whitelist ASN's based on organization name. This works by providing a list of ASN org names. This tool uses goPacket to monitor incoming traffic, capturing the IP's and checking the IP to see if it is a part of a whitelisted ASN. If it is not, it blocks that connection and future connections using iptables.

Jul 23, 2022
ptypes is a pointer-based box typing system for golang.

ptypes bypass go's type system through unsafe pointers the paradigm is to created a "boxed" type with .From and then use whatever types we want by ass

Aug 26, 2021
Fast, scalable pseudo random number generator based on xxh3
Fast, scalable pseudo random number generator based on xxh3

XXH3-Based Pseudorandom Number Generator This package contains an experimental implementation of a noise based pseudorandom number generator that scal

Nov 24, 2022
A concurrent rate limiter library for Golang based on Sliding-Window rate limiter algorithm.

ratelimiter A generic concurrent rate limiter library for Golang based on Sliding-window rate limitng algorithm. The implementation of rate-limiter al

Jan 6, 2023
Daypaper sets your GNOME wallpaper based on the time of day from a random and relevant Unsplash image.

Daypaper Daypaper sets your GNOME wallpaper based on the time of day from a random and relevant Unsplash image. Installation You will need an Access T

May 23, 2022
CUE utilities and helpers for working with tree based objects in any combination of CUE, Yaml, and JSON.

Cuetils CUE utilities and helpers for working with tree based objects in any combination of CUE, Yaml, and JSON. Using As a command line binary The cu

Dec 24, 2022
Configuration based URL shortner useful for on page 301 redirects

GO_URL_SHORT Configuration based URL shortner useful for on page 301 redirects Configuration for shortened url can be loaded from a JSON file in forma

Oct 16, 2021
A rule-based tunnel in Go with experimental features.
 A rule-based tunnel in Go with experimental features.

This repository is for archiving only Experimental-Clash A rule-based tunnel in Go with experimental features. Features Local HTTP/HTTPS/SOCKS server

Dec 3, 2021
The main goal of this code is to create a basic dnstap printing tool based on the golang-dnstap library.

dnstap-parse The main goal of this code is to create a basic dnstap printing tool based on the golang-dnstap library. The output is supposed to mimic

Nov 14, 2021
RoutePlanner suggests circular walks or runs based on start location and desired distance.
RoutePlanner suggests circular walks or runs based on start location and desired distance.

RoutePlanner Backend This repository contains code that powers the routeplanner app. The app suggests circular walks or runs based on start location a

Nov 5, 2021
Toolkits Codes Based on Go.

Toolkits Codes Based on Go.

Nov 2, 2022
GoLang-based client-side circuit breakers and helpers

Overview Example library for circuit breaking in GoLang. Written to support a blog post on https://www.wojno.com. Use this library in your SDK's to pr

Dec 5, 2021
Go 1.18 generics based slice and sorts.

genfuncs import "github.com/nwillc/genfuncs" Package genfuncs implements various functions utilizing Go's Generics to help avoid writing boilerplate c

Jan 2, 2023
Goety - Generics based Go utilities

goety General purpose Go utilities. Package channel Utilities to work with chann

May 16, 2022