ConfigMap Merging Controller (cmmc)
cmmc is a k8s operator that allows for the merging of ConfigMap resources with data validation.
Why?
The impetus for building this is to have a GitOps friendly solution to manage kube-system/aws-auth, a ConfigMap that binds AWS roles to K8S Roles in EKS (link).
Instead of solving the problem directly, our approach was to ask the question: If another tool existed that would make this problem trivial to solve that wasn't specific to this specific use-case, what would it be?
Features
- Watch specific keys of ConfigMaps and merge their results into a target.
- JSON Schema Validation for the target ConfigMap (possibly add other validation in the future).
- Fully Configurable source/target selectors mix and match
- Changes to resources are non-destructive and recoverable
- Permissions gated by namespace selectors (if desired)
- Metrics exposed for how many resources are being watched/updated, and their states.
How
The operator is built on the kubebuilder library and has two controllers/reconcilers, one for the MergeSource resource and one for MergeTarget.
Resources & Configuration
MergeSource
apiVersion: config.cmmc.k8s.cash.app/v1beta1
kind: MergeSource
metadata:
name: merge-map-roles-aws-auth
spec:
selector:
cmmc.k8s.cash.app/merge: "something"
source:
data: someKey
target:
name: our-merge-target
data: someKey
- A
MergeSourcedescribes whatConfigMapresource we are watching with itsselectorfield.
So anyConfigMapwith a label that matchesspec.selectorwill be watched. - The controller will read data from the
source.datafield on a matchingConfigMap - The
MergeSourcewill annotate the watched CMs so they know they are being watched. - This resource/controller does no mutatations of the data on any of the resources outside of the annotation!
- Annotations are cleaned up when the resource is deleted.
- The MergeTarget at
spec.target.namewill watch forMergeSourceresources with it as the target and read their aggregated states to attempt to write to the target ConfigMap.
MergeTarget
apiVersion: config.cmmc.k8s.cash.app/v1beta1
kind: MergeTarget
metadata:
name: our-merge-target
spec:
target: some-ns/some-resource-name # a configMap
data:
someKey:
init: ''
jsonSchema: |
{ … }
- A
MergeTargetdescribes the resource we are managing, in this case it issome-ns/some-resource-name. - We can configure this resource with the keys that we care about managing on the target, above its
someKey. - Each
data[$key]- Can have an initial value that we'll inject if the data was not present the key was missing or empty
- Can have an optional
jsonSchemathat we use to validate the data before it is persisted.
- Creates the ConfigMap if it doesn't exist.
- Uses annotations to make sure there is only one
MergeTargetperspec.target - Clean up after itself when it is deleted.
- If it didn't eist, it will be removed
- If it did exist, the data will be reset back to what it was before.
Demo (aws-auth)
Let's build a solution for managing kube-system/aws-auth.
1. Create a MergeTarget
A MergeTarget describes the target ConfigMap that want to write the data to.
cat <<EOF | kubectl apply -f -
apiVersion: config.cmmc.k8s.cash.app/v1beta1
kind: MergeTarget
metadata:
name: kube-system-aws-auth
spec:
target: kube-system/aws-auth
data:
mapRoles: {}
mapUsers: {}
EOF
This says that we want to write/merge data to the mapRoles and mapUsers keys of kube-system/aws-auth. Note, there is no auth, or initial value for these keys in this example, but we can add this later on.
2. Create a MergeSource for mapRoles
A MergeSource describes what ConfigMaps we are watching to write to the target. This one specifically looks for ConfigMap resources with the label: cmmc:k8s.cash.app/merge: "aws-auth-map-roles".
target.name refers to the MergeTarget we created earlier.
cat <<EOF | kubectl apply -f -
apiVersion: config.cmmc.k8s.cash.app/v1beta1
kind: MergeSource
metadata:
name: aws-auth-map-roles
spec:
selector:
cmmc.k8s.cash.app/merge: "aws-auth-map-roles"
source:
data: mapRoles
target:
name: kube-system-aws-auth
data: mapRoles
EOF
3. Create some sample ConfigMap sources
Let's create a sample configuration for two services/namespaces, service-a and service-b, which need some role binding from AWS to K8S.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
name: service-a
---
apiVersion: v1
kind: Namespace
metadata:
name: service-b
---
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-roles-mapping
namespace: service-a
labels:
cmmc.k8s.cash.app/merge: "aws-auth-map-roles"
data:
mapRoles: |
- arn: arn:aws:iam::111122223333:role/external-user-service-a
username: service-a-external
groups:
- service-a
---
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-roles-mapping
namespace: service-b
labels:
cmmc.k8s.cash.app/merge: "aws-auth-map-roles"
data:
mapRoles: |
- arn: arn:aws:iam::111122223333:role/external-user-service-b
username: service-b-external
groups:
- service-b
EOF
4. Check the resources
Target
kubectl get cm -n kube-system aws-auth -o yaml
apiVersion: v1
data:
mapRoles: |
- arn: arn:aws:iam::111122223333:role/external-user-service-b
username: service-b-external
groups:
- service-b
- arn: arn:aws:iam::111122223333:role/external-user-service-a
username: service-a-external
groups:
- service-a
kind: ConfigMap
metadata:
annotations:
config.cmmc.k8s.cash.app/managed-by-merge-target: default/kube-system-aws-auth
name: aws-auth
namespace: kube-system
Statuses
# kubectl get mergetarget
NAME TARGET READY STATUS VALIDATION
kube-system-aws-auth kube-system/aws-auth True Target ConfigMap up to date. 1 MergeSources reporting valid data
# kubectl get mergesource
NAME READY STATUS
aws-auth-map-roles True Data from 2 ConfigMap(s)
Cleanup
kubectl delete ns service-a
kubectl delete ns service-b
kubectl delete mergesource aws-auth-map-roles
kubectl delete mergetarget kube-system-aws-auth
