Kubernetes Response Engine powered by OpenFaaS
Although Falco can be used to detect any intrusion attempts and sends alerts to channels according to the given rules afterwards, it does not have any remediation system. This is why we need something called Kubernetes Response Engine. It simply aims to catch alerts and take actions on it. These actions can be designed as fine-grained serverless functions.
Think of a scenario you want to take action to your alerts that being notified by Falco, but the important thing that we should notice here is we are going to do different things for the different levels of alerts. For example, we are going to send the notification for notice level alert but for the warning level alert we are going to delete the pod, the overall architecture will be like this:
+-----------+
| Falco +
+-----^-----+
|
+-------v-------+
> OpenFaaS +
+-------v-------+
+-----------+ | +-----------+
| notify-fn <---------+----------> delete-fn |
+-----v-----+ notice warning +-----+-----+
| |
| send alert delete pod |
| |
+-----v-----+ +-----v-----+
| Slack | | Pwned Pod |
+-----------+ +-----------+
🎁
Table of Contents
Prerequisites
We are going to do this demo on macOS Catalina 1.15.7, you can find the prerequisites on brew.
Target Audience
If you want to:
- Set up a K3S cluster on CIVO
- Set up the Falco
- Create custom serverless functions using OpenFaaS
- Subscribe Falco events from serverless functions
What is ...?
CIVO
Built for speed and simplicity, with K3s under the hood
Join the #Kube100 beta: Apply to join today (Get free credit to test-drive the world’s first K3s-powered, managed Kubernetes service)
- Simple, fast and powerful
Spin up Kubernetes in under 2 minutes, without the bloat, using the lightweight K3s distribution
- Management CLI
Manage your clusters with the custom-built CLI and streamline your deployment with simple REST API.
- Application marketplace
Launch clusters with preinstalled applications, or install on the fly using Civo's Kubernetes marketplace.
- Bandwidth Pooling
Use all your bandwidth allowance across all your Civo services. Only pay for what you use.
- Marketplace
Install applications quickly and easily to your clusters during or after set up with Civo marketplace. Also you can contribute an application to marketplace
- Custom Firewalls
Configure custom firewalls from your Civo dashboard using OpenStack’s security groups.
K3S
The certified Kubernetes distribution built for IoT & Edge computing
- Perfect for Edge
K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances. Simplified & Secure
- Simplified & Secure
K3s is packaged as a single <40MB binary that reduces the dependencies and steps needed to install, run and auto-update a production Kubernetes cluster.
- Optimized for ARM
Both ARM64 and ARMv7 are supported with binaries and multiarch images available for both. K3s works great from something as small as a Raspberry Pi to an AWS a1.4xlarge 32GiB server.
Falco
Cloud-Native runtime security, de facto Kubernetes threat detection engine
- Strengthen container security
The flexible rules engine allows you to describe any type of host or container behavior or activity.
- Reduce risk via immediate alerts
You can immediately respond to policy violation alerts and integrate Falco within your response workflows.
- Leverage most current detection rules
Falco out-of-the box rules alert on malicious activity and CVE exploits.
OpenFaaS
OpenFaaS® makes it simple to deploy both functions and existing code to Kubernetes
- Anywhere
Avoid lock-in through the use of Docker. Run on any public or private cloud.
- Any code
Build both microservices & functions in any language. Legacy code and binaries.
- Any scale
Auto-scale for demand or to zero when idle.
Hands-on Demonstration
Create CIVO Playground
- Download the civo cli from here
- Copy and save the API key from your Civo account - Security Dashboard
$ civo apikey save my-awesome-key $KEY
Saved the API Key $KEY as my-awesome-key
$ civo apikey list
+----------------+---------+
| Name | Default |
+----------------+---------+
| my-awesome-key | $KEY |
+----------------+---------+
$ civo apikey current my-awesome-key
Set the default API Key to be my-awesome-key
- Create a cluster
$ civo kubernetes create playground --size=g3.k3s.medium --nodes=3 --region NYC1 --wait
Created Kubernetes cluster playground
- Show the playground on Kubernetes Dashboard
$ civo kubernetes show playground
ID : 79435efe-2dac-403d-bfd2-f6644988830a
Name : playground
Nodes : 3
Size : g3.k3s.medium
Status : ACTIVE
Version : 1.20.0-k3s2
API Endpoint : https://212.2.243.151:6443
Master IP : 212.2.243.151
DNS A record : 79435efe-2dac-403d-bfd2-f6644988830a.k8s.civo.com
Nodes:
+-------------+---------------+--------+---------------+-----------+------+----------+
| Name | IP | Status | Size | Cpu Cores | Ram | SSD disk |
+-------------+---------------+--------+---------------+-----------+------+----------+
| master-7c8a | 212.2.243.151 | ACTIVE | g3.k3s.medium | 2 | 4096 | 25 |
| node-04ed | | ACTIVE | g3.k3s.medium | 2 | 4096 | 25 |
| node-5258 | | ACTIVE | g3.k3s.medium | 2 | 4096 | 25 |
+-------------+---------------+--------+---------------+-----------+------+----------+
- Configure the playground
$ civo kubernetes config playground --save --local-path ./kubeconfig
Access your cluster with:
KUBECONFIG=./kubeconfig kubectl get node
- Ensure all is OK
$ export KUBECONFIG=./kubeconfig
$ kubectl get node
NAME STATUS ROLES AGE VERSION
k3s-playground-66b18d51-node-04ed Ready <none> 40h v1.20.2+k3s1
You can find more details about civo cli here.
- Set up OpenFaaS
You can install OpenFaaS from Civo marketplace -
or via helm
$ helm repo add openfaas https://openfaas.github.io/faas-netes/
$ kubectl create namespace openfaas-fn
$ helm upgrade openfaas --install openfaas/openfaas \
--namespace openfaas --create-namespace \
-f openfaas-override.yaml
Release "openfaas" does not exist. Installing it now.
NAME: openfaas
LAST DEPLOYED: Thu Mar 4 14:28:32 2021
NAMESPACE: openfaas
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that openfaas has started, run:
kubectl -n openfaas get deployments -l "release=openfaas, app=openfaas"
- Set up Falco
$ helm repo add falcosecurity https://falcosecurity.github.io/charts
$ helm upgrade falco --install falcosecurity/falco \
--namespace falco --create-namespace \
-f override.yaml
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: ./kubeconfig
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: ./kubeconfig
Release "falco" does not exist. Installing it now.
NAME: falco
LAST DEPLOYED: Thu Mar 4 14:34:54 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.
No further action should be required.
Tip:
You can easily forward Falco events to Slack, Kafka, AWS Lambda and more with falcosidekick.
Full list of outputs: https://github.com/falcosecurity/charts/falcosidekick.
You can enable its deployment with `--set falcosidekick.enabled=true` or in your values.yaml.
See: https://github.com/falcosecurity/charts/blob/master/falcosidekick/values.yaml for configuration values.
After everyting is ok, configure the faas-cli in order to deploy functions.
export OPENFAAS_URL="http://79435efe-2dac-403d-bfd2-f6644988830a.k8s.civo.com:31112"
Let's see what functions present.
$ faas-cli list
Function Invocations Replicas
You should see nothing.
NOTE: Don't forget to change Docker ID in the function's description files.
Now Deploy the functions.
$ faas-cli up -f dispatch-fn.yml
$ faas-cli up -f notifier-fn.yml
$ faas-cli up -f delete-pod-fn.yml
...
Once it's completed, you should see the pods that's running on openfaas-fn
namespace, and you should also see functions by typing faas-cli list
command.
$ faas-cli list
Function Invocations Replicas
dispatch-fn 37 1
notifier-fn 23 1
delete-pod-fn 0 1
$ kubectl get pods --namespace=openfaas-fn
Found existing alias for "kubectl get pods". You should use: "kgp"
NAME READY STATUS RESTARTS AGE
dispatch-fn-b5b68df77-r7crl 1/1 Running 0 7m23s
notifier-fn-5fd846c887-hgpf4 1/1 Running 0 4m48s
delete-pod-fn-55cc89965c-fs55q 1/1 Running 0 37s
If everything works well, you should see the Slack messages in the #falco-alert channel. Here is the screenshot above that shows this.
Learn more about how you can set up a Slack workspace from this link.
The next thing that you need to do this, you should create an alert at Warning level, in order to that, you need to create a Pod based on the alpine image, and you should connect to it, and access some important folders like /etc/shadow. Once this is done, you should notice that the Pod gets killed by the delete-pod-fn function.
Let's test it with creating a Pod.
$ kubectl run alpine -n default --image=alpine --restart='Never' -- sh -c "sleep 600"
Found existing alias for "kubectl". You should use: "k"
pod/alpine created
Connect to it.
$ kubectl exec -ti alpine -- sh
Found existing alias for "kubectl exec -ti". You should use: "keti"
/ #
Once you connect it, open a second terminal and watch the state of the Pod.
$ watch kubectl get pods
Every 2.0s: kubectl get pods
NAME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 106s
Then, in the first terminal, run the cat /etc/shadow
, then look at the second terminal, you should notice that Pod's status changed from Running to Terminating, and your connection to the Pod is lost.