Translate Prometheus Alerts into Kubernetes pod readiness

prometheus-alert-readiness

Translates firing Prometheus alerts into a Kubernetes readiness path.

Why?

By running this container in a singleton deployment with PriorityClass system-cluster-critical, we can prevent automated rolling-update tooling from proceeding (as they will not proceed when such high-priority pods are NotReady) when they shouldn't, due to firing Prometheus alerts when underlying databases are in e.g. under-replicated status.

Configuration

Configured by environment variables:

Variable Default Description
PROMETHEUS_ENDPOINT http://localhost:9090 The location of the Prometheus endpoint to send API requests to.
PROMETHEUS_API_TIMEOUT 10 How long the readiness check should wait for Prometheus to respond before timing out.
PROMETHEUS_ALERT_SEVERITIES critical,warning A comma-separated string of severities that will cause prometheus-alert-readiness to respond NotReady.
KUBE_LIVENESS_PATH /live The HTTP path on which the Kubernetes liveness probe will listen.
KUBE_READINESS_PATH /ready The HTTP path on which the Kubernetes readiness probe will listen.
KUBE_PROBE_LISTEN_PORT 8080 The HTTP port on which the prometheus-alert-readiness will listen.

Local dev

  1. Run docker build -t prometheus-alert-readiness . to build the container
  2. Expose the remote Prometheus host locally by running e.g. kubectl -n monitoring port-forward svc/kube-prometheus 9090
  3. Run docker run --rm --network host prometheus-alert-readiness:latest to run the container locally
  4. Run curl -i localhost:8080/ready to trigger a readiness check
Owner
Coralogix
A machine learning powered log analytics solution aimed to solve the problem of unmanageable log data
Coralogix
Similar Resources

SLOs, Error windows and alerts are complicated. Here an attempt to make it easy

slo-computer SLOs, Error windows and alerts are complicated. Here's an attempt to make it easy SLO, burn_rate, error_rate, budget_spend are convoluted

Dec 30, 2022

AWS Cloudtrail event alerting lambda function. Send alerts to Slack, Email, or SNS.

AWS Cloudtrail event alerting lambda function. Send alerts to Slack, Email, or SNS.

Cloudtrail-Tattletail is a Lambda based Cloudtrail alerting tool. It allows you to write simple rules for interesting Cloudtrail events and forward those events to a number of different systems.

Jan 6, 2023

A Terraform module that creates AWS alerts billing for your resources.

terraform-aws-billing-alarms terraform-aws-billing-alarms for project Replace name project to New Project agr 'terraform-aws-billing-alarms' 'new-pr

Oct 20, 2021

HSDP Metrics alerts webhook broker and CF events forwarder for Microsoft Teams

hsdp-events Microservice helper to translate HSDP Metrics webhooks to Microsoft Teams webhooks Configuration Environment Description EVENTS_TOKEN Rand

Mar 18, 2022

Alertmanager-cli is a cli writtin in golang to silence alerts in AlertManager

Alertmanager-cli is a cli writtin in golang to silence alerts in AlertManager

Aug 27, 2022

Export Prometheus metrics from journald events using Prometheus Go client library

journald parser and Prometheus exporter Export Prometheus metrics from journald events using Prometheus Go client library. For demonstration purposes,

Jan 3, 2022

A k8s vault webhook is a Kubernetes webhook that can inject secrets into Kubernetes resources by connecting to multiple secret managers

A k8s vault webhook is a Kubernetes webhook that can inject secrets into Kubernetes resources by connecting to multiple secret managers

k8s-vault-webhook is a Kubernetes admission webhook which listen for the events related to Kubernetes resources for injecting secret directly from sec

Oct 15, 2022

Exporter your cypress.io dashboard into prometheus Metrics

Cypress.io dashboard Prometheus exporter Prometheus exporter for a project from Cypress.io dashboards, giving the ability to alert, make special opera

Feb 8, 2022

General Pod Autoscaler(GPA) is a extension for K8s HPA, which can be used not only for serving, also for game.

General Pod Autoscaler(GPA) is a extension for K8s HPA, which can be used not only for serving, also for game.

Introduction General Pod Autoscaler(GPA) is a extension for K8s HPA, which can be used not only for serving, also for game. Features Compatible with a

Aug 19, 2022
Comments
  • Create Dependabot config file

    Create Dependabot config file

    :wave: Dependabot is moving natively into GitHub! This pull request migrates your configuration from Dependabot.com to a config file, using the new syntax. When you merge this pull request, we'll swap out dependabot-preview (me) for a new dependabot app, and you'll be all set!

    With this change, you'll now use the Dependabot page in GitHub, rather than the Dependabot dashboard, to monitor your version updates. Dependabot is now configured exclusively using config files.

    Your account is using config variables to access private registries. GitHub-native Dependabot will support private registries before Dependabot Preview is shut down. For now, we recommend leaving Dependabot Preview active.

    If you've got any questions or feedback for us, please let us know by creating an issue in the dependabot/dependabot-core repository.

    Learn more about the relaunch of Dependabot

    Please note that regular @dependabot commands do not work on this pull request.

    :robot::yellow_heart:

A kubernetes plugin which enables dynamically add or remove GPU resources for a running Pod
A kubernetes plugin which enables dynamically add or remove GPU resources for a running Pod

GPU Mounter GPU Mounter is a kubernetes plugin which enables add or remove GPU resources for running Pods. This Introduction(In Chinese) is recommende

Jan 5, 2023
⎈ Multi pod and container log tailing for Kubernetes

stern Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging. T

Nov 7, 2022
A Kubernetes Mutating Webhook to automatically re-point pod images to mirrors

kubernetes-mimic Kubernetes Mimic is a Mutating Webhook that will watch for pod creation and update events in a Kubernetes cluster and automatically a

Nov 22, 2022
An example of Kubernetes' Horizontal Pod Autoscaler using costume metrics.
An example of Kubernetes' Horizontal Pod Autoscaler using costume metrics.

Kubernetes Autoscaling Example In this project, I try to implement Horizontal Pod AutoscalerHPA provided by Kubernetes. The Horizontal Pod Autoscaler

Dec 1, 2022
A docker container that can be deployed as a sidecar on any kubernetes pod to monitor PSI metrics

CgroupV2 PSI Sidecar CgroupV2 PSI Sidecar can be deployed on any kubernetes pod with access to cgroupv2 PSI metrics. About This is a docker container

Nov 23, 2021
Kubernetes Pod Security Standards implementation

Pod Security Admission The Pod Security Standards are a set of best-practice profiles for running pods securely. This repository contains the codified

Dec 30, 2022
Kubernetes create pod cli with golang

kubernatestest- Anand #Added more coments Anand2 and K8s #GO COMMANDS to RUN : go run xxx.go to install : go install xyz.go - this created binary file

Dec 2, 2021
gpupod is a tool to list and watch GPU pod in the kubernetes cluster.

gpupod gpupod is simple tool to list and watch GPU pod in kubernetes cluster. usage Usage: gpupod [flags] Flags: -t, --createdTime with pod c

Dec 8, 2021
A kubernetes operator sample generated by kubebuilder , which run cmd in pod on specified time

init kubebuilder init --domain github.com --repo github.com/tonyshanc/sample-operator-v2 kubebuilder create api --group sample --version v1 --kind At

Jan 25, 2022
A very simple, silly little kubectl plugin / utility that guesses which language an application running in a kubernetes pod was written in.

A very simple, silly little kubectl plugin / utility that guesses which language an application running in a kubernetes pod was written in.

Mar 9, 2022