Rotate is a tool for rotating out AWS Auto-Scaling Groups within a k8s cluster

k8s-r8

rotate is a tool for rotating out AWS Auto-Scaling Groups within a k8s cluster. It was developed to make upgrading AMIs as a one command experience that doesn't require intimate knowledge of AWS commands or Kubernetes internals.

Assumptions

  1. All nodes in the cluster have a role
  2. Each role used within the cluster has a 1:1 mapping with an AWS ASG that manages all of the nodes in that role
  3. None of the pods running in your cluster maintain persistent volumes
  4. The name of each node in your cluster is the internal DNS name of the node in AWS

Requirements

  1. An AWS Profile where the default entry has permissions to modify ASGs
  2. A Kubernetes config that points to the cluster you wish to rotate. The config's user must have privileges to cordon nodes, delete pods, evict pods, and list the nodes in the cluster.

Usage

This tool is built via a simple go build cmd/rotate/main.go or go install cmd/rotate

Ensure none of the target nodes have SchedulingDisabled status or the script will block until the node is manually kubectl uncordoned.

Examples

rotate

by default the rotate command will find all of the roles in your k8s cluster, and rotate a single node in the ASG corresponding to each role. The default behavior is to rotate the oldest node in the cluster by launch configuration and then by age.

rotate --rotate-all

This will find all of the roles in your k8s cluster, and incrementally rotate all of the nodes in those ASGs. ASG need to be the auto scaling group name, eg staging-20211015165147895500000001. You can find this in AWS console -> ec2 -> auto scaling group role name needs to be the kubernetes role, eg ingress or compute. You can find this with kubectl describe node.

rotate --asg-to-role=ASG:ROLENAME --rotate-all
AWS_PROFILE=twitch-safety-staging AWS_REGION=ap-southeast-3 go run cmd/rotate/main.go --rotate-all --asg-to-role='safety-staging-ingress-nodes-20211015165147895500000001:ingress'

This will find the ASG with the given name, make sure it corresponds to the specified role, and if so, rotate all nodes in that ASG incrementally. Removing the --rotate-all will lead to the tool rotating a single node.

Flags

-f/--strict-delete - when this flag is enabled, if there are any errors during pod deletion or eviction while spinning down a node, the tool will error out. Otherwise, these errors will be logged and then ignored. The default is that this flag is off.

-e/--evict-grace-period - argument - duration - the default value is 5 seconds. This flag controls how long we will wait for a pod to be deleted/evicted before raising an error. This is useful if your pod has a long shutdown protocol.

-b/--network-backoff - argument - duration - the default value is 5 seconds. This flag controls the time between requests when the tool is polling the kubernetes cluster to see if a new node has come up or an old node has gone down.

-r/--asg-to-role - argument map:ASG-NAME:cluster-role - This takes a mapping of asg names to cluster roles. For each pair, the tool will rotate the oldest node in the cluster. If the -u/--rotate-all flag is enabled, then all the nodes in all of the asg/cluster pairs will be rotated. If only one pair is provided, this can be combined with the -n/--node flag to specify a list of nodes to rotate in the asg. If no asgs are specified, the tool will search through all ASGs in the target AWS account, and try to correspond them with a role in the cluster. For each mapping that can be determined, the tool will rotate one (or if -u/--rotate-all is specified, all) of the nodes those asg/role groups.

-u/--rotate-all - Disabled by default. When enabled, this makes it so that the tool rotates all of the nodes in any asg:role pairings that are targeted.

-n/--node- argument, list of k8s node names - This REQUIRES a single ASG:Role pairing be provided via -r/--asg-to-role. This flag will make the tool rotate all of the nodes specified in this list.

Owner
maikxchd
certified bubble sort hater
maikxchd
Similar Resources

K8s-ingress-health-bot - A K8s Ingress Health Bot is a lightweight application to check the health of the ingress endpoints for a given kubernetes namespace.

k8s-ingress-health-bot A K8s Ingress Health Bot is a lightweight application to check the health of qualified ingress endpoints for a given kubernetes

Jan 2, 2022

K8s-go-structs - All k8s API Go structs

k8s-api go types Why? Its nice to have it all in a single package. . |-- pkg |

Jul 17, 2022

🔭 Kubernetes out-cluster vulnerability scanner

🔭 Kubernetes out-cluster vulnerability scanner

Kubnerable Kubnerable is an out-cluster vulnerability scanner tool for Kubernetes resources. It comes with a predefined vulnerability database (vulner

Mar 26, 2022

⚙️ Operating Account Operators (OAO) is a Golang tool to interact with the LDAP protocol to manage account groups, roles, ACLs/ACEs, etc...

⚙️ Operating Account Operators (OAO) is a Golang tool to interact with the LDAP protocol to manage account groups, roles, ACLs/ACEs, etc...

⚙️ OAO (Operating Account Operators) ⚙️ Operating Account Operators (OAO) is a Golang tool to interact with the LDAP protocol to manage account groups

May 11, 2023

CoreDNS plugin implementing K8s multi-cluster services DNS spec.

corends-multicluster Name multicluster - implementation of Multicluster DNS Description This plugin implements the Kubernetes DNS-Based Multicluster S

Dec 3, 2022

Manages nodes in hybrid k8s self-hosted cluster

node-manager Manages nodes in hybrid k8s self-hosted cluster Supported providers Contabo Hetzner Robot (dedicated) Supported commands Heal - reboots a

Dec 23, 2021

Kubelet-bench - Example Go-based e2e benchmark for various Kubelet operations without spinning up whole K8s cluster

kubelet-bench An example of Go based e2e benchmark for various Kubelet operation

Mar 17, 2022

A kubectl plugin for easier query and operate k8s cluster.

A kubectl plugin for easier query and operate k8s cluster.

kube-query A kubectl plug-in that makes it easier to query and manipulate K8S clusters. (what is kubectl plug-in ?) Kube-query support some resource s

Jun 9, 2022

A Kubernetes Operator used for pre-scaling applications in anticipation of load

Pre-Scaling Kubernetes Operator Built out of necessity, the Operator helps pre-scale applications in anticipation of load. At its core, it manages a c

Oct 14, 2021
Related tags
Create AWS Auto Scaling groups from running instances or AMIs automatically.

AWS Auto Scaling Groups Builder AWS Auto Scaling group is a great way of managing Amazon EC2 instances. AWS Auto Scaling group watches the correspondi

Oct 7, 2022
Service-scaling - Microservice scaling with example

Scaling examples This is the repo for my (Hungarian) Twitch streams where I spea

Jan 8, 2022
A Terraform module to manage cluster authentication (aws-auth) for an Elastic Kubernetes (EKS) cluster on AWS.

Archive Notice The terraform-aws-modules/eks/aws v.18.20.0 release has brought back support aws-auth configmap! For this reason, I highly encourage us

Dec 4, 2022
kubetnl tunnels TCP connections from within a Kubernetes cluster to a cluster-external endpoint, e.g. to your local machine. (the perfect complement to kubectl port-forward)

kubetnl kubetnl (kube tunnel) is a command line utility to tunnel TCP connections from within a Kubernetes to a cluster-external endpoint, e.g. to you

Dec 16, 2022
A pod scaler golang app that can scale replicas either inside of cluster or out of the cluster

pod-scaler A simple pod scaler golang application that can scale replicas via manipulating the deployment Technologies The project has been created us

Oct 24, 2021
K8s controller implementing Multi-Cluster Services API based on AWS Cloud Map.

AWS Cloud Map MCS Controller for K8s Introduction AWS Cloud Map multi-cluster service discovery for Kubernetes (K8s) is a controller that implements e

Dec 17, 2022
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds

Website • Quickstart • Documentation • Blog • Twitter • Slack vcluster - Virtual Clusters For Kubernetes Lightweight & Low-Overhead - Based on k3s, bu

Jan 4, 2023
Git-auto-push - Auto commit and push to github repositories

Auto commit and push to github repositories. How to use git clone https://github

Dec 19, 2022
Write controller-runtime based k8s controllers that read/write to git, not k8s

Git Backed Controller The basic idea is to write a k8s controller that runs against git and not k8s apiserver. So the controller is reading and writin

Dec 10, 2021
K8s-cinder-csi-plugin - K8s Pod Use Openstack Cinder Volume

k8s-cinder-csi-plugin K8s Pod Use Openstack Cinder Volume openstack volume list

Jul 18, 2022