metrics-agent
The metrics-agent collects allocation metrics from a Kubernetes cluster system and sends the metrics to cloudability to help you gain visibility, reduce costs, and increase efficiency across your infrastructure. The agent is designed to run as a container in each cluster inside your orchestration system.
Kubernetes
By default, the agent runs in a namespace named "cloudability" (see options below). Once deployed, the agent will pull metrics from the Kubernetes API and directly from each node in the cluster it is running in. Additionally it will pull metrics from Heapster if found running in the kube-system namespace. An example kubernetes deployment can be found here.
Every 10 minutes the metrics agent creates a tarball of the gathered metrics and uploads to an Amazon Web Service S3 bucket. This process requires outbound connections to https://metrics-collector.cloudability.com/, to obtain a pre-signed URL, and https://cldy-cake-pipeline.s3.amazonaws.com/ to upload the data. If the metrics agent is deployed behind a firewall, these addresses should be added to the outbound allow list.
Supported Versions
1.21 and below
Kubernetes versions 1.21 and below are supported by the metrics agent.
Configuration Options
| Environment Variable | Description |
|---|---|
| CLOUDABILITY_API_KEY | Required: Cloudability api key |
| CLOUDABILITY_CLUSTER_NAME | Required: The cluster name to be used for the cluster the agent is running in. |
| CLOUDABILITY_POLL_INTERVAL | Optional: The interval (Seconds) to poll metrics. Default: 180 |
| CLOUDABILITY_HEAPSTER_URL | Optional: Only required if heapster is not deployed as a service in your cluster or is only accessable via a specific URL. |
| CLOUDABILITY_OUTBOUND_PROXY | Optional: The URL of an outbound HTTP/HTTPS proxy for the agent to use (eg: http://x.x.x.x:8080). The URL must contain the scheme prefix (http:// or https://) |
| CLOUDABILITY_OUTBOUND_PROXY_AUTH | Optional: Basic Authentication credentials to be used with the defined outbound proxy. If your outbound proxy requires basic authentication credentials can be defined in the form username:password |
| CLOUDABILITY_OUTBOUND_PROXY_INSECURE | Optional: When true, does not verify TLS certificates when using the outbound proxy. Default: False |
| CLOUDABILITY_INSECURE | Optional: When true, does not verify certificates when making TLS connections. Default: False |
| CLOUDABILITY_RETRIEVE_NODE_SUMMARIES | Optional: When true, collects metrics directly from each node in a cluster. When False, uses Heapster as the primary metrics source. Default: True |
| CLOUDABILITY_GET_ALL_CONTAINER_STATS | Optional: When true, attempts to collect from both the stats/container and metrics/cadvisor endpoints, which may result in a larger metrics payload. When False, only collects first successful endpoint. Default: False |
| CLOUDABILITY_FORCE_KUBE_PROXY | Optional: When true, forces agent to use the proxy to connect to nodes rather than attempting a direct connection. Default: False |
| CLOUDABILITY_COLLECT_HEAPSTER_EXPORT | Optional: When true, attempts to collect metrics from Heapster if available. When False, does not collect Heapster metrics. Default: True |
| CLOUDABILITY_COLLECTION_RETRY_LIMIT | Optional: Number of times agent should attempt to gather metrics from each source upon a failure Default: 1 |
| CLOUDABILITY_NAMESPACE | Optional: Override the namespace that the agent runs in. It is not recommended to change this as it may negatively affect the agents ability to collect data. Default: cloudability |
| CLOUDABILITY_LOG_FORMAT | Optional: Format for log output (JSON,PLAIN) Default: PLAIN |
| CLOUDABILITY_LOG_LEVEL | Optional: Log level to run the agent at (INFO,WARN,DEBUG,TRACE). Default: INFO |
| CLOUDABILITY_SCRATCH_DIR | Optional: Temporary directory that metrics will be written to. If set, must assure that the directory exists and that the user agent UID 1000 has read/write access to the folder. Default: /tmp |
metrics-agent kubernetes --help
Command to collect Kubernetes Metrics
Usage:
metrics-agent kubernetes [flags]
Flags:
--api_key string Cloudability API Key - required
--certificate_file string The path to a certificate file. - Optional
--cluster_name string Kubernetes Cluster Name - required this must be unique to every cluster.
--heapster_override_url string URL to connect to a running heapster instance. - optionally override the discovered Heapster URL.
--collection_retry_limit uint Number of times agent should attempt to gather metrics from each source upon a failure (default 1)
-h, --help help for kubernetes
--insecure When true, does not verify certificates when making TLS connections. Default: False
--key_file string The path to a key file. - Optional
--outbound_proxy string Outbound HTTP/HTTPS proxy eg: http://x.x.x.x:8080. Must have a scheme prefix (http:// or https://) - Optional
--outbound_proxy_auth string Outbound proxy basic authentication credentials. Must defined in the form username:password - Optional
--outbound_proxy_insecure When true, does not verify TLS certificates when using the outbound proxy. Default: False
--retrieve_node_summaries When true, collects metrics directly from each node in a cluster. When False, uses Heapster as the primary metrics source. Default: True
--get_all_container_stats When true, attempts to collect from both the stats/container and metrics/cadvisor endpoints, which may result in a larger metrics payload. Default: False
--force_kube_proxy When true, forces agent to use the proxy to connect to nodes rather than attempting a direct connection. Default: False
--poll_interval int Time, in seconds, to poll the services infrastructure. Default: 180 (default 180)
--namespace string The namespace which the agent runs in. Changing this is not recommended. (default `cloudability`)
Global Flags:
--log_format string Format for log output (JSON,PLAIN) (default "PLAIN")
--log_level string Log level to run the agent at (INFO,WARN,DEBUG) (default "INFO")
Development
Dependency management
We're using go modules for Go dependencies.
Source Code Analysis
We're using golangci-lint for static source code analysis.
Contributing code
You'll find information and help on how to contribute code in the CONTRIBUTING document in this repo.
To Run Locally
You must obtain a valid API Key and export it locally as an environment variable.
export CLOUDABILITY_API_KEY={your_api_key}
make deploy-local
Local Development
The makefile target deploy-local assumes that you have docker and kubernetes (with a context: docker-for-desktop) running locally. The target does the following:
- Builds a container with the local project codebase
- Locally creates a deployment / pod with the local metrics agent container
Testing
In addition to running all go tests via the make step make test, make test-e2e-all runs end to end tests by spinning up a kind cluster, building the metrics agent, deploying it to the reference clusters, then testing the collected data. The use of kind requires a local docker daemon to be running.
