An example of Kubernetes' Horizontal Pod Autoscaler using costume metrics.

Kubernetes Autoscaling Example

In this project, I try to implement Horizontal Pod AutoscalerHPA provided by Kubernetes. The Horizontal Pod Autoscaler automatically scales the number of Pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics).

We start by creating a golang web-server that publishes messages to a rabbitmq queue. These messages are then consumed by a NodeJS worker. The worker takes 5s to consume 1 message, representing artificial time to process a request. We then try to scale the worker pods on the basis of the rabbitmq queue length. We'll discuss the entire infra in the upcoming section and role of each part in detail.

Demo

Architecture


Web-server

A simple golan server with two routes, one for pushing messages to rabbitmq queue and one for exposing metrics. We are collecting 2 metrics, http_request_total and request_status to monitor number of request on and status returned by each route respectively.

Routes

GET /generate    # push message to queue
GET /metrics     # expose metrics for Prometheus server

RabbitMQ

Rabbit MQ is a simple open-source message-broker that can be deployed with ease. The broker has wide support of client libraries across different programming languages. Rabbitmq is widely used in industry and has proven it's mettle. We use Advance Message Queuing Protocol(AMQP) as our messaging protocol for both web-server and worker.

NodeJS Worker

A simple nodejs application that connects to rabbitmq queue anc consumes messages. In order to generate a real life scenario, we take one message at a time and add a delay of 5 sec, representing the time needed to process a message. This way we'll be able to generate fake traffic when we try to run our architecture.

Prometheus-stack

As discussed above, our goal is to auto-scale by looking at the queue length anf to achieve this we need to add monitoring service to our Cluster. Prometheus is one of the best options available for monitoring tasks. To make things easy and save ourselves from writing manifest files, we'll use helm charts for deployment. We use a kube-prometheus-stack chart by prometheus-community. The chart comes with Prometheus, Grafana, Prometheus operator and lot of default configurations for these services. Lets look at the role of each component.

  • Prometheus: a open-sourced CNCF graduate responsible for all our monitoring and alerting needs across the cluster. I was originally built by SoundCloud and then handed over to CNCF for maintenance and further development.

  • Grafana: a open source observability platform using generate beautifully dashboard. We use Grafana to consume Prometheus metrics in a human readable fashion.

  • Prometheus Operator: The Prometheus Operator provides Kubernetes native deployment and management of Prometheus and related monitoring components. The purpose of this project is to simplify and automate the configuration of a Prometheus based monitoring stack for Kubernetes cluster.

RabbitMQ exporter

Most of the service does not support Prometheus metrics out of box and need a middleware to do so. This is where Exporters come in picture. In our case, rabbitmq's management service collects metrics but they are not compliant for Prometheus. There are community is actively adding exports and thus it likely that you'll find one for your service. You can check for different available exporter here. We use prometheus-rabbitmq-exporter helm chart to make deployment process easy. Now Prometheus server can pull metrics from this exporter.

Prometheus Adapter

Kubernetes only monitors recourses like compute and memory metrics for pods. While defining our HPA manifest we need to mention the type of metric to look for. There are three types of metrics available

  • Resource: default monitoring by Kubernetes
  • Custom: metrics that are not recorded by k8s but is related to it's Objects
  • External: metrics that are not related to k8s Objects.

We use External type as Queue length is not monitored by Kubernetes and is not related to Kubernetes Objects. Since all objects talk to API-server, adapter exposes these metrics to API server.

How to run

In this example, we are using minikube as our local cluster setup. We start by deploying our Kube-prometheus-stack first. We'll use the default config values for this setup.

Building docker images

You can use docker-compose command to build the web-server and Worker node images as they'll be used in cluster. To build images simply run docker-compose build


$ helm install prometheus prometheus-community/kube-prometheus-stack

Next deploy our RabbitMQ server, Web-Server and NodeJs worker using manifest files.

$ kubectl apply -f rabbitmq/deploy.yml
$ kubectl apply -f web-server/deploy.yml
$ kubectl apply -f worker/deploy.yml

Once our application is up, we proceed with rabbitmq exporter and prometheus adapter.

$ helm install rabbitmq-exporter prometheus-community/prometheus-rabbitmq-exporter -f rabbitmq/values.yml
$ helm install prometheus-adapter prometheus-community/prometheus-adapter -f prometheus/adapter-values.yml

So your now our prometheus adapter is up lets see if our metrics is exposed.

$ kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/" | jq

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "rabbitmq_queue_length",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

As seen above, the metric is available and is ready for API server to consume. Now the final setup is to create our HPA. We create a HPA for our worker using manifest file that is available under worker/hpa.yml

$ kubectl apply -f worker/hpa.yml
horizontalpodautoscaler.autoscaling/worker-scaler created

$ kubectl get hpa
NAME            REFERENCE           TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
worker-scaler   Deployment/worker   0/20      1         10        1          22s

So are HPA is created and is ready to scale our worker deployment. You can see that the current Replica count is 1 since the metric value is under target. Now you can expose your web-server service using kubectl port-forward command and hit /generate route from browser multiple time in order to generate some traffic. You'll see the metric value rise and Replica count increase.

My learnings

Initially I considered it to be a easy and started very casually, but soon I realised different accepts of the problem. I didn't know that I'll have to use Prometheus as a metric server cause rabbitmq's management image exposed the need metrics on /api/metrics route. So I was in impression that I'll just need to pass pass this url to HPA and use the metric to scale. I was wrong( or didn't get a way to achieve this), thats not how Kubernetes work. I got to know about why we need to use Prometheus to use metrics, then I learnt about exporters and there role in this metric capturing setup. Once I was ready with my metric, next challenge was to get to HPA. After some thorough search I got to know about Prometheus Adapter and it's role in the architecture. Once Adapter was in it's place, I had to play with HPA and different Adapater config in order to understand it's working. I tried all three kinds of HPA, Resource,Custom and External and understood when to use which type.

Was fun building such small project to get insights of the Kubernetes and it's working. I will continue building such project in order to learn more and more about it. I would like to thank Daniele Polencic whose talk motivated to design this architecture.

Similar Resources

A Kubernetes Mutating Webhook to automatically re-point pod images to mirrors

kubernetes-mimic Kubernetes Mimic is a Mutating Webhook that will watch for pod creation and update events in a Kubernetes cluster and automatically a

Nov 22, 2022

Kubernetes Pod Security Standards implementation

Pod Security Admission The Pod Security Standards are a set of best-practice profiles for running pods securely. This repository contains the codified

Dec 30, 2022

Kubernetes create pod cli with golang

kubernatestest- Anand #Added more coments Anand2 and K8s #GO COMMANDS to RUN : go run xxx.go to install : go install xyz.go - this created binary file

Dec 2, 2021

gpupod is a tool to list and watch GPU pod in the kubernetes cluster.

gpupod gpupod is simple tool to list and watch GPU pod in kubernetes cluster. usage Usage: gpupod [flags] Flags: -t, --createdTime with pod c

Dec 8, 2021

A kubernetes operator sample generated by kubebuilder , which run cmd in pod on specified time

init kubebuilder init --domain github.com --repo github.com/tonyshanc/sample-operator-v2 kubebuilder create api --group sample --version v1 --kind At

Jan 25, 2022

A very simple, silly little kubectl plugin / utility that guesses which language an application running in a kubernetes pod was written in.

A very simple, silly little kubectl plugin / utility that guesses which language an application running in a kubernetes pod was written in.

Mar 9, 2022

Flash-metrics - Flash Metrics Storage With Golang

Flash Metrics Storage bootstrap: $ echo -e "max-index-length = 12288" tidb.con

Jan 8, 2022

This library provides a metrics package which can be used to instrument code, expose application metrics, and profile runtime performance in a flexible manner.

This library provides a metrics package which can be used to instrument code, expose application metrics, and profile runtime performance in a flexible manner.

Jan 18, 2022

Pulumi-k8s-operator-example - OpenGitOps Compliant Pulumi Kubernetes Operator Example

Pulumi GitOps Example OpenGitOps Compliant Pulumi Kubernetes Operator Example Pr

May 6, 2022
The metrics-agent collects allocation metrics from a Kubernetes cluster system and sends the metrics to cloudability

metrics-agent The metrics-agent collects allocation metrics from a Kubernetes cluster system and sends the metrics to cloudability to help you gain vi

Jan 14, 2022
A docker container that can be deployed as a sidecar on any kubernetes pod to monitor PSI metrics

CgroupV2 PSI Sidecar CgroupV2 PSI Sidecar can be deployed on any kubernetes pod with access to cgroupv2 PSI metrics. About This is a docker container

Nov 23, 2021
cluster-api-state-metrics (CASM) is a service that listens to the Kubernetes API server and generates metrics about the state of custom resource objects related of Kubernetes Cluster API.

Overview cluster-api-state-metrics (CASM) is a service that listens to the Kubernetes API server and generates metrics about the state of custom resou

Oct 27, 2022
Sensu-go-postgres-metrics - The sensu-go-postgres-metrics is a sensu check that collects PostgreSQL metrics

sensu-go-postgres-metrics Table of Contents Overview Known issues Usage examples

Jan 12, 2022
Annotated and kubez-autoscaler-controller will maintain the HPA automatically for kubernetes resources.

Kubez-autoscaler Overview kubez-autoscaler 通过为 deployment / statefulset 添加 annotations 的方式,自动维护对应 HorizontalPodAutoscaler 的生命周期. Prerequisites 在 kuber

Jan 2, 2023
DigitalOcean Droplets target plugin for HashiCorp Nomad Autoscaler

Nomad DigitalOcean Droplets Autoscaler The do-droplets target plugin allows for the scaling of the Nomad cluster clients via creating and destroying D

Dec 8, 2022
Example Pod webhook

Pod Webhook Example Local Development # Create a local cluster. kind create cluster # Setup cluster dependencies (cert-manager). ./hack/setup.sh # D

Nov 30, 2021
A kubernetes plugin which enables dynamically add or remove GPU resources for a running Pod
A kubernetes plugin which enables dynamically add or remove GPU resources for a running Pod

GPU Mounter GPU Mounter is a kubernetes plugin which enables add or remove GPU resources for running Pods. This Introduction(In Chinese) is recommende

Jan 5, 2023
Translate Prometheus Alerts into Kubernetes pod readiness

prometheus-alert-readiness Translates firing Prometheus alerts into a Kubernetes readiness path. Why? By running this container in a singleton deploym

Oct 31, 2022
⎈ Multi pod and container log tailing for Kubernetes

stern Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging. T

Nov 7, 2022