kubequery is a Osquery extension that provides SQL based analytics for Kubernetes clusters

Build CodeQL Go Report Card FOSSA Status Contributor Covenant


kubequery powered by Osquery

kubequery is a Osquery extension that provides SQL based analytics for Kubernetes clusters

kubequery will be packaged as docker image available from dockerhub. It is expected to be deployed as a Kubernetes Deployment per cluster. A sample deployment template is available here

kubequery tables schema is available here


Build

Go 1.15 and make are required to build kubequery. Run: make

Container image for master branch will be available on dockerhub

docker pull uptycs/kubequery:latest

For production, tagged container images should be used instead of latest.


Deployment

kubequery.yaml is a template that creates the following Kubernetes resources:

kubequery Namespace will be the placeholder for all resources that are namespaced.

kubequery-sa is ServiceAccount that is associated with the kubequery deployment pod specification. The container uses the service account token to authenticate with the API server.

kubequery-clusterrole is a ClusterRole that allows get and list operations on all resources in the following API groups:

  • "" (core)
  • admissionregistration.k8s.io
  • apps
  • autoscaling
  • batch
  • networking.k8s.io
  • policy
  • rbac.authorization.k8s.io
  • storage.k8s.io

kubequery-clusterrolebinding is a ClusterRoleBinding that binds the cluster role with the service account.

kubequery-config is a ConfigMap that will be mounted inside the container image as a directory. The contents of this config map should be similar to /etc/osquery. For example, osquery.flags, osquery.conf, etc. should be part of this config map.

kubequery is the Deployment that creates one replica pod. The container launched as a part of the pod is run as non-root user.

By default pod resource requests and limits are set to 500m (half a core) and 200MB. kubequery.yaml file should be tweaked to suite your needs before applying:

kubectl apply -f kubequery.yaml

Validate the installation was successful by first executing:

kubectl exec -it $(kubectl get pods -n kubequery -o jsonpath='{.items[0].metadata.name}') -- bash -c 'osqueryi --extension /usr/bin/kubequery'

and then running

.tables kubernetes

Which should produce the following output:

  => kubernetes_api_resources
  => kubernetes_cluster_role_binding_subjects
  => kubernetes_cluster_role_policy_rule
  => kubernetes_config_maps
  => kubernetes_cron_jobs
  => kubernetes_csi_drivers
  => kubernetes_csi_node_drivers
  => kubernetes_daemon_set_containers
  ...

FAQ

Kubernetes events support?

kubenetes_events table can be easily implemented in kubequery as traditional table. But ideally it should be a streaming events table similar to process_events etc in Osquery. Unfortunately Osquery does not support event tables in extensions currently. Buffering the data in extension and periodically sending it in response to a query is one option, but it is not ideal.

Use kubequery instead of Osquery in Kubernetes?

No. kubequery should to be deployed as a Kubernetes Deployment. Which means there will be one Pod of kubequery running per Kubernetes cluster. Osquery should be deployed to every node in the cluster. Querying most Osquery tables from an ephemeral pod does not provide much value. kubequery container image also runs as non-root user, which means most of the Osquery tables will either return an error or partial data.

Deployment

Why are some columns JSON?

Normalizing nested JSON data like Kubernetes API responses will create an explosion of tables. So some of the columns in kuberenetes tables are left as JSON. Data is eventually processed by SQLite with-in Osquery. SQLite has very good JSON support.

For example if run_as_user in kubernetes_pod_security_policies table looks like the following:

{"rule": "MustRunAsNonRoot"}

To get the value of rule, the following query can be used:

SELECT value AS 'rule'
FROM kubernetes_pod_security_policies, json_tree(kubernetes_pod_security_policies.run_as_user)
WHERE key = 'rule';

+------------------+
| rule             |
+------------------+
| MustRunAsNonRoot |
+------------------+

json_each can be used to explode JSON array types. For example if volumes in kubernetes_pod_security_policies table looks like the following:

{"volumes": ["configMap","emptyDir","projected","secret","downwardAPI","persistentVolumeClaim"]}

To get a separate row for each volume, the following query can be used:

SELECT value
FROM kubernetes_pod_security_policies, json_each(kubernetes_pod_security_policies.volumes);

+-----------------------+
| value                 |
+-----------------------+
| configMap             |
| emptyDir              |
| projected             |
| secret                |
| downwardAPI           |
| persistentVolumeClaim |
+-----------------------+

Osquery logger's like TLS, Kafka loggers can be used to export scheduled query data to remove fleet management/security analytics platforms. Lamba like functions can be applied on rows of streaming data in these platforms. These lamba functions can extract necessary fields from embedded JSON to detect compliance issues or security concerns. If tables are normalized and are streamed at different schedules, it will not be trivial to JOIN across tables and trigger events/alerts.

Owner
Uptycs Inc
Uptycs Inc
Uptycs Inc
Comments
  • Startup fails with panic: extension ping failed

    Startup fails with panic: extension ping failed

    Describe the bug

    kubectl -n kubequery logs kubequery-7b768854-pkrwf
    panic: extension ping failed: read unix @->/opt/kubequery/osquery.em: read: connection reset by peer
    
    goroutine 1 [running]:
    main.main()
    	/home/runner/work/kubequery/kubequery/cmd/kubequery/main.go:57 +0x489
    W0124 01:10:59.180693     7 watcher.cpp:557] osqueryd worker respawning too quickly: 1 times
    W0124 01:11:05.407027   239 extensions.cpp:781] Required extension not found or not loaded: kubequery
    E0124 01:11:05.407080   239 init.cpp:553] An error occurred during extension manager startup: Required extension not found or not loaded: kubequery
    I0124 01:11:05.408862   239 events.cpp:867] Event publisher not enabled: BPFEventPublisher: Publisher disabled via configuration
    I0124 01:11:05.408974   239 events.cpp:867] Event publisher not enabled: auditeventpublisher: Publisher disabled via configuration
    I0124 01:11:05.409005   239 events.cpp:867] Event publisher not enabled: inotify: Publisher disabled via configuration
    I0124 01:11:05.409034   239 events.cpp:867] Event publisher not enabled: syslog: Publisher disabled via configuration
    

    To Reproduce Apply the provided yaml (https://github.com/Uptycs/kubequery/blob/master/kubequery.yaml) on a Docker Desktop k8s cluster.

    Expected behavior Startup succeeds.

    Screenshots If applicable, add screenshots to help explain your problem.

    Additional context latest kubequery image

  • Slow JOIN

    Slow JOIN

    Describe the bug I run this query "select c.uid from kubernetes_pod_containers c, kubernetes_pods p where p.uid = c.uid limit 1;" on 2000 pods , 3000 containers and it hangs, or takes forever.

    To Reproduce run the query

    Expected behavior should be fast since the JOIN is on primary keys.

    Screenshots If applicable, add screenshots to help explain your problem.

    Additional context

    osquery 5.0.2.0 using SQLite 3.35.5

  • Check container status before iterating over contents

    Check container status before iterating over contents

    By submitting a PR to this repository, you agree to the terms within the Code of Conduct. Please see the contributing guidelines for how to create and submit a PR for this repo.

    Description

    Check container status before iterating over contents

    References

    Testing

  • Query kubernetes_pod_containers table failed

    Query kubernetes_pod_containers table failed

    Describe the bug query kubernetes_pod_containers table failed, error is Extension call failed: No more data to read

    To Reproduce run SELECT * FROM kubernetes_pod_containers;

    Expected behavior k8s containers should be returned

    Screenshots

    panic: runtime error: index out of range [0] with length 0
    
    goroutine 198 [running]:
    github.com/Uptycs/kubequery/internal/k8s/core.PodContainersGenerate(0x1a4b7f0, 0xc000046048, 0xc000380960, 0x0, 0x0, 0x0, 0xc000089b10, 0x40fc38)
    	/home/runner/work/kubequery/kubequery/internal/k8s/core/pod.go:133 +0xb05
    github.com/Uptycs/basequery-go/plugin/table.(*Plugin).Call(0xc0001cf830, 0x1a4b7f0, 0xc000046048, 0xc000380270, 0xc00025f3b8, 0xc000688001, 0x0, 0xc000089b88)
    	/home/runner/go/pkg/mod/github.com/!uptycs/[email protected]/plugin/table/table.go:68 +0x366
    github.com/Uptycs/basequery-go.(*ExtensionManagerServer).Call(0xc000160360, 0x1a4b7b8, 0xc002210240, 0xc003b3a061, 0x5, 0xc0010c6000, 0x19, 0xc000380270, 0x6ea9dc, 0xc0004684d0, ...)
    	/home/runner/go/pkg/mod/github.com/!uptycs/[email protected]/server.go:254 +0x10f
    github.com/Uptycs/basequery-go/gen/osquery.(*extensionProcessorCall).Process(0xc00021de10, 0x1a4b7b8, 0xc002210240, 0x0, 0x1a67ca8, 0xc0004684d0, 0x1a67ca8, 0xc000468540, 0x176b600, 0x0, ...)
    	/home/runner/go/pkg/mod/github.com/!uptycs/[email protected]/gen/osquery/osquery.go:1449 +0x1c6
    github.com/Uptycs/basequery-go/gen/osquery.(*ExtensionProcessor).Process(0xc00000e0c0, 0x1a4b860, 0xc0001cedb0, 0x1a67ca8, 0xc0004684d0, 0x1a67ca8, 0xc000468540, 0xc0001cedb0, 0x7f34383b7aa8, 0x419c37)
    	/home/runner/go/pkg/mod/github.com/!uptycs/[email protected]/gen/osquery/osquery.go:1311 +0x3b4
    github.com/apache/thrift/lib/go/thrift.(*TSimpleServer).processRequests(0xc00014e140, 0x1a56b78, 0xc0001ce510, 0x0, 0x0)
    	/home/runner/go/pkg/mod/github.com/apache/[email protected]/lib/go/thrift/simple_server.go:316 +0x31e
    github.com/apache/thrift/lib/go/thrift.(*TSimpleServer).innerAccept.func1(0xc00014e140, 0x1a56b78, 0xc0001ce510)
    	/home/runner/go/pkg/mod/github.com/apache/[email protected]/lib/go/thrift/simple_server.go:198 +0x6f
    created by github.com/apache/thrift/lib/go/thrift.(*TSimpleServer).innerAccept
    	/home/runner/go/pkg/mod/github.com/apache/[email protected]/lib/go/thrift/simple_server.go:196 +0x15f
    W0723 07:13:11.395772    15 watcher.cpp:602] Extension respawning too quickly: /opt/uptycs/bin/kubequery.ext
    I0723 07:13:11.396102    15 watcher.cpp:636] Created and monitoring extension child (19639): /opt/uptycs/bin/kubequery.ext
    I0723 07:13:11.458142    49 extensions.cpp:348] Extension UUID 36288 has gone away
    

    Additional context kubelet_version: v1.18.8 kubeproxy_version: v1.18.8 container_runtime_version: docker://18.6.1

  • Additional Guidance on Install

    Additional Guidance on Install

    By submitting a PR to this repository, you agree to the terms within the Code of Conduct. Please see the contributing guidelines for how to create and submit a PR for this repo.

    Description

    I saw this on https://cncn.io and I had a hard time getting it started. I've added comments that helped me figure out how to interact with the osqueryi shell.

    References

    Not Applicable

    Testing

    Tested commands against a Digital Ocean cluster running kubernetes v1.19.3.

  • Update README.md

    Update README.md

    By submitting a PR to this repository, you agree to the terms within the Code of Conduct. Please see the contributing guidelines for how to create and submit a PR for this repo.

    Description

    Update README.md to refer to Go 1.17

    References

    Include any links supporting this change such as:

    • GitHub Issue/PR number addressed or fixed
    • Related pull requests/issues from other repos

    Testing

    Describe the testing done. Setup required for testing. Details of unit or other tests added as a part of the PR.

  • Upgrade Golang dependencies. Use staticcheck for lint

    Upgrade Golang dependencies. Use staticcheck for lint

    By submitting a PR to this repository, you agree to the terms within the Code of Conduct. Please see the contributing guidelines for how to create and submit a PR for this repo.

    Description

    Upgrade Golang dependencies. Use staticcheck for lint

    References

    None

    Testing

    Local

  • Readme Updates

    Readme Updates

    Signed-off-by: Travis Lowe [email protected]

    By submitting a PR to this repository, you agree to the terms within the Code of Conduct. Please see the contributing guidelines for how to create and submit a PR for this repo.

    Description

    Fixed a couple of broken links, and updated the jsonpath output values to include quotes required for it to work.

  • Generate tables.json

    Generate tables.json

    By submitting a PR to this repository, you agree to the terms within the Code of Conduct. Please see the contributing guidelines for how to create and submit a PR for this repo.

    Description

    Generate tables.json similar to Osquery

    References

    Include any links supporting this change such as:

    • GitHub Issue/PR number addressed or fixed
    • Related pull requests/issues from other repos

    Testing

    Describe the testing done. Setup required for testing. Details of unit or other tests added as a part of the PR.

  • DET-0000: Add on demand run for codeql

    DET-0000: Add on demand run for codeql

    By submitting a PR to this repository, you agree to the terms within the Code of Conduct. Please see the contributing guidelines for how to create and submit a PR for this repo.

    Description

    Describe the purpose of this PR along with any background information and the impacts of the proposed change.

    References

    Include any links supporting this change such as:

    • GitHub Issue/PR number addressed or fixed
    • Related pull requests/issues from other repos

    Testing

    Describe the testing done. Setup required for testing. Details of unit or other tests added as a part of the PR.

  • Clean container_id and image_id. Add separate column from image_repo which is the image registry and image name

    Clean container_id and image_id. Add separate column from image_repo which is the image registry and image name

    By submitting a PR to this repository, you agree to the terms within the Code of Conduct. Please see the contributing guidelines for how to create and submit a PR for this repo.

    Description

    Clean container_id and image_id. These are now just SHA values which prefix. Add separate column from image_repo which is the image registry and image name

    References

    Include any links supporting this change such as:

    • GitHub Issue/PR number addressed or fixed
    • Related pull requests/issues from other repos

    Testing

    Describe the testing done. Setup required for testing. Details of unit or other tests added as a part of the PR.

  • support remote access

    support remote access

    Describe the bug allow kubequery to access remote clusters

    To Reproduce Steps to reproduce the behavior.

    Expected behavior allow kubequery to access remote clusters to support 'aggregator' mode.

    Screenshots If applicable, add screenshots to help explain your problem.

    Additional context kubequery version Kubernetes cluster information Add any other context about the problem here.

    Made these changes to k8s/client.go to support this 'kubequery aggregator' function : https://github.com/christiancadieux/kubequery-postgres

    diff --git a/internal/k8s/client.go b/internal/k8s/client.go
    index d2d2081..9b6942a 100644
    --- a/internal/k8s/client.go
    +++ b/internal/k8s/client.go
    @@ -16,6 +16,7 @@ import (
     	"os"
     	"path/filepath"
     	"sync"
    +	"time"
     
     	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
     	"k8s.io/apimachinery/pkg/types"
    @@ -32,7 +33,31 @@ var (
     	clusterName string
     )
     
    +func UserConfig(token, addr string) *rest.Config{
    +
    +	timeout := 10
    +
    +	tlsConfig := rest.TLSClientConfig{Insecure: true}
    +
    +	kubeConfig := &rest.Config{
    +		Host:            addr,
    +		BearerToken:     token,
    +		TLSClientConfig: tlsConfig,
    +		QPS:             250,
    +		Burst:           1000,
    +		Timeout:         time.Duration(timeout) * time.Second,
    +	}
    +
    +	return kubeConfig
    +}
    +
    +
     func initClientset(config *rest.Config) error {
    +	var err error
    +	if os.Getenv("KQ_TOKEN") != "" && os.Getenv("KQ_ADDR") != "" {
    +		config = UserConfig(os.Getenv("KQ_TOKEN"), os.Getenv("KQ_ADDR"))
    +	}
    +
     	if config == nil {
     		conf, err := rest.InClusterConfig()
     		if err != nil {
    @@ -52,7 +77,6 @@ func initClientset(config *rest.Config) error {
     	// Suppress deprecation warnings
     	config.WarningHandler = rest.NoWarnings{}
     
    -	var err error
     	clientset, err = kubernetes.NewForConfig(config)
     	if err != nil {
     		return err
    @@ -66,6 +90,12 @@ func initUID() error {
     		return err
     	}
     
    +	if os.Getenv("KQ_CLUSTER_NAME") != "" &&  os.Getenv("KQ_CLUSTER_UID") != ""{
    +		clusterName = os.Getenv("KQ_CLUSTER_NAME")
    +		clusterUID = types.UID(os.Getenv("KQ_CLUSTER_UID"))
    +		return nil
    +	}
    +
     	clusterUID = ks.UID
    
    
KinK is a helper CLI that facilitates to manage KinD clusters as Kubernetes pods. Designed to ease clusters up for fast testing with batteries included in mind.
KinK is a helper CLI that facilitates to manage KinD clusters as Kubernetes pods. Designed to ease clusters up for fast testing with batteries included in mind.

kink A helper CLI that facilitates to manage KinD clusters as Kubernetes pods. Table of Contents kink (KinD in Kubernetes) Introduction How it works ?

Dec 10, 2022
Client extension for interacting with Kubernetes clusters from your k6 tests.

⚠️ This is a proof of concept As this is a proof of concept, it won't be supported by the k6 team. It may also break in the future as xk6 evolves. USE

Jan 2, 2023
cloudquery powered by Osquery

cloudquery powered by Osquery cloudquery is Osquery extension to fetch cloud telemetry from AWS, GCP, and Azure. It is extensible so that one can add

Dec 25, 2022
Fleet - Open source device management, built on osquery.
Fleet - Open source device management, built on osquery.

Fleet - Open source device management, built on osquery.

Dec 30, 2022
Kubernetes OS Server - Kubernetes Extension API server exposing OS configuration like sysctl via Kubernetes API

KOSS is a Extension API Server which exposes OS properties and functionality using Kubernetes API, so it can be accessed using e.g. kubectl. At the moment this is highly experimental and only managing sysctl is supported. To make things actually usable, you must run KOSS binary as root on the machine you will be managing.

May 19, 2021
In this repository, the development of the gardener extension, which deploys the flux controllers automatically to shoot clusters, takes place.

Gardener Extension for Flux Project Gardener implements the automated management and operation of Kubernetes clusters as a service. Its main principle

Dec 3, 2022
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds

Website • Quickstart • Documentation • Blog • Twitter • Slack vcluster - Virtual Clusters For Kubernetes Lightweight & Low-Overhead - Based on k3s, bu

Jan 4, 2023
Kubernetes IN Docker - local clusters for testing Kubernetes
Kubernetes IN Docker - local clusters for testing Kubernetes

kind is a tool for running local Kubernetes clusters using Docker container "nodes".

Jan 5, 2023
provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters

provider-kubernetes provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters ty

Dec 14, 2022
Crossplane provider to provision and manage Kubernetes objects on (remote) Kubernetes clusters.

provider-kubernetes provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters ty

Jan 3, 2023
Kubernetes IN Docker - local clusters for testing Kubernetes
Kubernetes IN Docker - local clusters for testing Kubernetes

Please see Our Documentation for more in-depth installation etc. kind is a tool for running local Kubernetes clusters using Docker container "nodes".

Feb 14, 2022
PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes.

GalaxyKube -- PolarDB-X Operator PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes. It follo

Dec 19, 2022
Terraform module to provisison Kubernetes Clusters on Hetzner cloud (Based on KubeOne)

Terraform module template Terraform module which creates describe your intent resources on AWS. Usage Use this template to scaffold a new terraform mo

Nov 26, 2021
KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
 KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes

Kubernetes-based Event Driven Autoscaling KEDA allows for fine-grained autoscaling (including to/from zero) for event driven Kubernetes workloads. KED

Jan 7, 2023
A GitHub CLI extension that provides summary pull request metrics.

gh-metrics A gh extension that provides summary pull request metrics. Usage Metric definitions Influences Usage To install the extension use: $ gh ext

Dec 29, 2022
🐶 Kubernetes CLI To Manage Your Clusters In Style!
🐶 Kubernetes CLI To Manage Your Clusters In Style!

K9s - Kubernetes CLI To Manage Your Clusters In Style! K9s provides a terminal UI to interact with your Kubernetes clusters. The aim of this project i

Jan 9, 2023
Validation of best practices in your Kubernetes clusters
Validation of best practices in your Kubernetes clusters

Best Practices for Kubernetes Workload Configuration Fairwinds' Polaris keeps your clusters sailing smoothly. It runs a variety of checks to ensure th

Jan 9, 2023
Manage large fleets of Kubernetes clusters
Manage large fleets of Kubernetes clusters

Introduction Fleet is GitOps at scale. Fleet is designed to manage up to a million clusters. It's also lightweight enough that it works great for a si

Dec 31, 2022
A best practices checker for Kubernetes clusters. 🤠

Clusterlint As clusters scale and become increasingly difficult to maintain, clusterlint helps operators conform to Kubernetes best practices around r

Dec 29, 2022