Prometheus-elasticache-sd - Prometheus Service Discovery for AWS ElastiCache

Prometheus AWS ElastiCache Service Discovery

build

ElastiCache SD allows retrieving scrape targets from AWS ElastiCache cache nodes for Prometheus. No address is defined by default, it must be configured with relabeling and requires a third-party exporter supporting the multi-target pattern.

Configuration

Help on flags:

./prometheus-elasticache-sd --help

The following meta labels are available on targets during relabeling:

  • __meta_elasticache_cache_cluster_id: The identifier of the cluster.
  • __meta_elasticache_cache_cluster_status: The current state of this cluster. See CacheClusterStatus in the API Reference for possible values.
  • __meta_elasticache_cache_node_id: The cache node identifier. A node ID is a numeric identifier (0001, 0002, etc.). The combination of cluster ID and node ID uniquely identifies every cache node used in a customer's Amazon account.
  • __meta_elasticache_cache_node_status: The current state of this cache node. See CacheNodeStatus in the API Reference for possible values.
  • __meta_elasticache_cache_node_type: The name of the compute and memory capacity node type for the cluster.
  • __meta_elasticache_cache_parameter_group_name: The name of the cache parameter group.
  • __meta_elasticache_cache_subnet_group_name: The name of the cache subnet group associated with the cluster.
  • __meta_elasticache_customer_availability_zone: The Availability Zone where this node was created and now resides.
  • __meta_elasticache_endpoint_address: The DNS hostname of the cache node.
  • __meta_elasticache_endpoint_port: The port number that the cache engine is listening on.
  • __meta_elasticache_engine_version: The version of the cache engine that is used in this cluster.
  • __meta_elasticache_engine: The name of the cache engine (memcached or redis) used for this cluster.
  • __meta_elasticache_preferred_availability_zone: The name of the Availability Zone in which the cluster is located or "Multiple" if the cache nodes are located in different Availability Zones.
  • __meta_elasticache_replication_group_id: The replication group to which this cluster belongs. If this label is absent, the cluster is not associated with any replication group.
  • __meta_elasticache_tag_<tagkey>: The tag's value.

The following AWS IAM permissions are required:

  • elasticache:DescribeCacheClusters
  • elasticache:ListTagsForResource

Usage

Docker

To run the ElastiCache SD as a Docker container, run:

docker run ghcr.io/maxbrunet/prometheus-elasticache-sd:latest --help

oliver006/redis_exporter

This service discovery can be used with oliver006/redis_exporter, here is a sample Prometheus configuration:

scrape_configs:
  - job_name: "redis_exporter_targets"
    file_sd_configs:
    - files:
        - /path/to/elasticache.json  # Set file path with --output.file flag
    metrics_path: /scrape
    relabel_configs:
      # Filter for Redis cache nodes
      - source_labels: [__meta_elasticache_engine]
        regex: redis
        action: keep
      # Build Redis URL to use as target parameter for the exporter
      - source_labels:
          - __meta_elasticache_endpoint_address
          - __meta_elasticache_endpoint_port
        replacement: redis://$1
        separator: ':'
        target_label: __param_target
      # Use Redis URL as instance label
      - source_labels: [__param_target]
        target_label: instance
      # Set exporter address
      - target_label: __address__
        replacement: <<REDIS-EXPORTER-HOSTNAME>>:9121

Development

Build

docker build -t prometheus-elasticache-sd .

Test

go test -v ./...

License

Apache License 2.0, see LICENSE.

Comments
  • Cluster Discovery for single Cache Cluster ID

    Cluster Discovery for single Cache Cluster ID

    Hello, First of all, thanks for this project. It is a great idea to be able to automatically fetch endpoints from cluster nodes automatically.

    I'm having an issue. Currently I have several clusters with password, which means I'm forced to deploy one redis_exporter per each cluster, because redis_exporter only works with a single password for all the provided targets. This is fine, a bit more resource consumption but not a big deal. Because of that, I was trying to setup the prometheus-elasticache-sd to work with a single cache cluster (--elasticache.cache-cluster-id) but I just realized that the AWS API is returning each redis node as its own cluster. The reason is that we actually have ReplicationGroups for redis, which creates a cache-cluster for each node (apparently).

    Let's assume I have an elasticache with the following properties:

    • name: my-elasticache-redis
    • nodes: 2 (1 primary, 1 read replica)
    • shards: 1

    I'd expect the prometheus-elasticache-sd to receive this name "my-elasticache-redis" and return the endpoints from the 2 nodes. But it will say that there is no cache cluster named "my-elasticache-redis". When I query the cache clusters with the AWS API, I see 2 cache-clusters: "my-elasticache-redis-001" and "my-elasticache-redis-002"... Which defeats the purpose of autodiscovering the nodes. If I set --elasticache.cache-cluster-id=my-elasticache-redis-001 it works but only registers the endpoint from one of the nodes, not both.

    What are your thoughts around adding support for ReplicationGroups?

  • fix: fix nodes in cache cluster overwriting each other's labels

    fix: fix nodes in cache cluster overwriting each other's labels

    The previous code was using the same instance of LabelSet for creating labels for all cache nodes, this would override all cache nodes to use the same labelset as the last one.

    This solution simply clones one instance of the CacheCluster label set to each of the nodes before overriding it.

  • build(deps): update aws-sdk-go-v2 monorepo

    build(deps): update aws-sdk-go-v2 monorepo

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/aws/aws-sdk-go-v2/config | require | patch | v1.18.6 -> v1.18.7 | | github.com/aws/aws-sdk-go-v2/credentials | require | patch | v1.13.6 -> v1.13.7 |


    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

    👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

  • build(deps): update module github.com/prometheus/prometheus to v0.41.0

    build(deps): update module github.com/prometheus/prometheus to v0.41.0

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/prometheus/prometheus | require | minor | v0.40.7 -> v0.41.0 |


    Release Notes

    prometheus/prometheus

    v0.41.0

    Compare Source


    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

  • chore(deps): update actions/cache action to v3.2.2

    chore(deps): update actions/cache action to v3.2.2

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | actions/cache | action | patch | v3.2.1 -> v3.2.2 |


    Release Notes

    actions/cache

    v3.2.2

    Compare Source

    What's Changed

    New Contributors

    Full Changelog: https://github.com/actions/cache/compare/v3.2.1...v3.2.2


    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

  • build(deps): update aws-sdk-go-v2 monorepo

    build(deps): update aws-sdk-go-v2 monorepo

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | Pending | |---|---|---|---|---| | github.com/aws/aws-sdk-go-v2/config | require | patch | v1.18.5 -> v1.18.6 | v1.18.7 | | github.com/aws/aws-sdk-go-v2/credentials | require | patch | v1.13.5 -> v1.13.6 | v1.13.7 |


    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

    👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

  • chore(deps): update actions/cache action to v3.2.1

    chore(deps): update actions/cache action to v3.2.1

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | actions/cache | action | patch | v3.2.0 -> v3.2.1 |


    Release Notes

    actions/cache

    v3.2.1

    Compare Source

    What's Changed

    Full Changelog: https://github.com/actions/cache/compare/v3.2.0...v3.2.1


    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

  • build(deps): update aws-sdk-go-v2 monorepo

    build(deps): update aws-sdk-go-v2 monorepo

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | Pending | |---|---|---|---|---| | github.com/aws/aws-sdk-go-v2 | require | patch | v1.17.2 -> v1.17.3 | | | github.com/aws/aws-sdk-go-v2/config | require | patch | v1.18.4 -> v1.18.5 | v1.18.7 (+1) | | github.com/aws/aws-sdk-go-v2/credentials | require | patch | v1.13.4 -> v1.13.5 | v1.13.7 (+1) | | github.com/aws/aws-sdk-go-v2/service/elasticache | require | patch | v1.24.2 -> v1.24.3 | | | github.com/aws/aws-sdk-go-v2/service/sts | require | patch | v1.17.6 -> v1.17.7 | |


    Release Notes

    aws/aws-sdk-go-v2

    v1.17.3

    Compare Source


    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

    👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

  • chore(deps): update actions/cache action to v3.2.0

    chore(deps): update actions/cache action to v3.2.0

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | actions/cache | action | minor | v3.0.11 -> v3.2.0 |


    Release Notes

    actions/cache

    v3.2.0

    Compare Source

    What's Changed

    New Contributors

    Full Changelog: https://github.com/actions/cache/compare/v3...v3.2.0


    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

  • build(deps): update module github.com/prometheus/common to v0.39.0

    build(deps): update module github.com/prometheus/common to v0.39.0

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/prometheus/common | require | minor | v0.38.0 -> v0.39.0 |


    Release Notes

    prometheus/common

    v0.39.0

    Compare Source

    • [ENHANCEMENT] Add support for proxy connect headers. #​409
    • [ENHANCEMENT] Add platform info to labels. #​403

    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

  • build(deps): update module github.com/prometheus/prometheus to v0.40.7

    build(deps): update module github.com/prometheus/prometheus to v0.40.7

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | Pending | |---|---|---|---|---| | github.com/prometheus/prometheus | require | patch | v0.40.6 -> v0.40.7 | v0.41.0 |


    Release Notes

    prometheus/prometheus

    v0.40.7

    Compare Source


    Configuration

    📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

    🔕 Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

  • Discovery plugin

    Discovery plugin

    Prometheus now supports built-time plugins.

    It would be cool to convert the main package to a discovery plugin so that it could be built-in to Prometheus.

  • Dependency Dashboard

    Dependency Dashboard

    This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

    Pending Status Checks

    These updates await pending status checks. To force their creation now, click the checkbox below.

    • [ ] build(deps): update module github.com/aws/aws-sdk-go-v2/service/elasticache to v1.25.0

    Detected dependencies

    dockerfile
    Dockerfile
    • golang 1.19.4-alpine@sha256:a9b24b67dc83b3383d22a14941c2b2b2ca6a103d805cac6820fd1355943beaf1
    • quay.io/prometheus/busybox sha256:4ee0a8dced91db2c03eb2a69f20e95a41ad9602ebf3e6892b2666c02cf99cc78
    github-actions
    .github/workflows/build.yml
    • actions/checkout v3.2.0@755da8c3cf115ac066823e79a1e1788f8940201b
    • actions/setup-go v3.5.0@6edd4406fa81c3da01a34fa6f6343087c207a568
    • actions/cache v3.2.2@4723a57e26efda3a62cbde1812113b730952852d
    • actions/checkout v3.2.0@755da8c3cf115ac066823e79a1e1788f8940201b
    • actions/setup-go v3.5.0@6edd4406fa81c3da01a34fa6f6343087c207a568
    • actions/cache v3.2.2@4723a57e26efda3a62cbde1812113b730952852d
    • golangci/golangci-lint-action v3.3.1@0ad9a0988b3973e851ab0a07adf248ec2e100376
    • actions/checkout v3.2.0@755da8c3cf115ac066823e79a1e1788f8940201b
    • github/codeql-action v2.1.37@959cbb7472c4d4ad70cdfe6f4976053fe48ab394
    • github/codeql-action v2.1.37@959cbb7472c4d4ad70cdfe6f4976053fe48ab394
    • github/codeql-action v2.1.37@959cbb7472c4d4ad70cdfe6f4976053fe48ab394
    • actions/checkout v3.2.0@755da8c3cf115ac066823e79a1e1788f8940201b
    • docker/setup-buildx-action v2.2.1@8c0edbc76e98fa90f69d9a2c020dcb50019dc325
    • docker/build-push-action v3.2.0@c56af957549030174b10d6867f20e78cfd7debc5
    • actions/checkout v3.2.0@755da8c3cf115ac066823e79a1e1788f8940201b
    • docker/setup-buildx-action v2.2.1@8c0edbc76e98fa90f69d9a2c020dcb50019dc325
    • docker/login-action v2.1.0@f4ef78c080cd8ba55a85445d5b36e214a81df20a
    • cycjimmy/semantic-release-action v3.2.0@8f6ceb9d5aae5578b1dcda6af00008235204e7fa
    • docker/build-push-action v3.2.0@c56af957549030174b10d6867f20e78cfd7debc5
    gomod
    go.mod
    • go 1.19
    • github.com/aws/aws-sdk-go-v2 v1.17.3
    • github.com/aws/aws-sdk-go-v2/config v1.18.7
    • github.com/aws/aws-sdk-go-v2/credentials v1.13.7
    • github.com/aws/aws-sdk-go-v2/service/elasticache v1.24.3
    • github.com/aws/aws-sdk-go-v2/service/sts v1.17.7
    • github.com/aws/smithy-go v1.13.5
    • github.com/go-kit/log v0.2.1
    • github.com/prometheus/client_golang v1.14.0
    • github.com/prometheus/common v0.39.0
    • github.com/prometheus/exporter-toolkit v0.8.2
    • github.com/prometheus/prometheus v0.41.0
    • gopkg.in/alecthomas/kingpin.v2 v2.2.6
    regex
    Dockerfile
    • github.com/prometheus/promu v0.14.0
    .github/workflows/build.yml
    • github.com/golangci/golangci-lint v1.50.1
    • semantic-release 19.0.5
    .go-version
    • go 1.19.4
    .github/workflows/build.yml
    • conventional-changelog-conventionalcommits 5.0.0
    • @semantic-release/changelog 6.0.2
    • @semantic-release/git 10.0.1

    • [ ] Check this box to trigger a request for Renovate to run again on this repository
Rpcx-framework - An RPC microservices framework based on rpcx, simple and easy to use, ultra fast and efficient, powerful, service discovery, service governance, service layering, version control, routing label registration.

RPCX Framework An RPC microservices framework based on rpcx. Features: simple and easy to use, ultra fast and efficient, powerful, service discovery,

Jan 5, 2022
A service registry and service discovery implemention for kitex based on etcd

kitex etcd Introduction kitexetcd is an implemention of service registry and service discovery for kitex based on etcd. Installation go get -u github.

Feb 18, 2022
A package for access aws service using AWS SDK for Golang

goaws ?? A package for access aws service using AWS SDK for Golang Advantage with goaws package Example for get user list IAM with AWS SDK for Golang

Nov 25, 2021
Use Consul to do service discovery, use gRPC +kafka to do message produce and consume. Use redis to store result.
Use  Consul to do service discovery, use gRPC +kafka to do message produce and consume. Use redis to store result.

目录 gRPC/consul/kafka简介 gRPC+kafka的Demo gRPC+kafka整体示意图 限流器 基于redis计数器生成唯一ID kafka生产消费 kafka生产消费示意图 本文kafka生产消费过程 基于pprof的性能分析Demo 使用pprof统计CPU/HEAP数据的

Jul 9, 2022
The mec platform for service register/discovery/subscribe and other functions.roject main repo.

EdgeGallery MEP project Introduction Edgegallery MEP is an open source implementation of MEC platform according to ETSI MEC 003 [1] and 011 [2] docume

Nov 15, 2022
Nacos is a service config & discovery.

Kratos Nacos example sc := []constant.ServerConfig{ *constant.NewServerConfig("127.0.0.1", 8848), } cc := &constant.ClientConfig{ NamespaceId:

Oct 28, 2021
Polaris is a cloud-native service discovery and governance center

It can be used to solve the problem of service connection, fault tolerance, traffic control and secure in distributed and microservice architecture.

Dec 26, 2022
Service registration and discovery, support etcd, zookeeper, consul, etc.

discox 支持类型 zookeeper etcd consul 示例 zookeeper server package main import ( "fmt" "github.com/goeasya/discox" "os" ) func main() { cfg := discox

Aug 31, 2022
Service Discovery and Governance Center for Distributed and Microservice Architecture
Service Discovery and Governance Center for Distributed and Microservice Architecture

Polaris: Service Discovery and Governance English | 简体中文 README: Introduction Components Getting started Chat group Contribution Visit website to lear

Dec 31, 2022
Jun 20, 2022
Envoy-eds-server - Envoy EDS server is a working Envoy Discovery Service implementation

envoy-eds-server Intro Envoy EDS server is a working Envoy Discovery Service imp

Apr 2, 2022
DNS service discovery library for Go

Discovery DNS service discovery library for Go Documentation see pkg.go.dev Installation

Mar 10, 2022
Service registry/discovery implementation in Go.

go-service-registry Availabe endpoints : GET http://localhost:3000/ --> Dashboard GET http://localhost:3000/services/[serviceName] --> Get available

Feb 11, 2022
Simple no frills AWS S3 Golang Library using REST with V4 Signing (without AWS Go SDK)

simples3 : Simple no frills AWS S3 Library using REST with V4 Signing Overview SimpleS3 is a golang library for uploading and deleting objects on S3 b

Nov 4, 2022
Run the same Docker images in AWS Lambda and AWS ECS
Run the same Docker images in AWS Lambda and AWS ECS

serverlessish tl;dr Run the exact same image for websites in Lambda as you do in ECS, Kubernetes, etc. Just add this to your Dockerfile, listen on por

Dec 22, 2022
CLI for exploring AWS EC2 Spot inventory. Inspect AWS Spot instance types, saving, price, and interruption frequency.

spotinfo The spotinfo is a command-line tool that helps you determine AWS Spot instance types with the least chance of interruption and provides the s

Dec 19, 2022
This plugin helps you to use the AWS Command Line Interface (AWS CLI) to start and end sessions to your managed instances

Session Manager Plugin This plugin helps you to use the AWS Command Line Interface (AWS CLI) to start and end sessions to your managed instances. Sess

Dec 28, 2022
Terraform provider to help with various AWS automation tasks (mostly all that stuff we cannot accomplish with the official AWS terraform provider)
Terraform provider to help with various AWS automation tasks (mostly all that stuff we cannot accomplish with the official AWS terraform provider)

terraform-provider-awsutils Terraform provider for performing various tasks that cannot be performed with the official AWS Terraform Provider from Has

Dec 8, 2022
Infrastructure testing helper for AWS Resources that uses AWS SSM to remotely execute commands on EC2 machines.
Infrastructure testing helper for AWS Resources that uses AWS SSM to remotely execute commands on EC2 machines.

Infrastructure testing helper for AWS Resources that uses AWS SSM to remotely execute commands on EC2 machines, to enable infrastructure engineering teams to write tests that validate behaviour.

Sep 5, 2022