Currently Cortex only support GCP, AWS Dynamo, Cassandra for Chunk Index. Here add a new choice, which is ElasticSearch since it's a very popular NoSQL storage.

Implement Hint: Only write some new go code in pkg/chunk/elastic, other file changes are introduced by go mod tidy and go mod vendor.

Sample Config to Use ElasticSearch in https addr(will skip tls verify by default, user can change it with passing some config) and http addr:

schema_config:
        configs:
        - from: 2018-04-15
          store: elastic
          object_store: s3
          schema: v9
          index:
            prefix: index_
            period: 168h

      storage_config:
        elastic:
          address: https://es_addr:443
          user: user
          password: password

or

schema_config:
        configs:
        - from: 2018-04-15
          store: elastic
          object_store: s3
          schema: v9
          index:
            prefix: index_
            period: 168h

      storage_config:
        elastic:
          address: https://es_addr:443
          user: user
          password: password
          tls_skip_verify: false
          cert_file: cert_file_addr
          key_file: key_file_addr
          ca_file: ca_file

and

schema_config:
        configs:
        - from: 2018-04-15
          store: elastic
          object_store: s3
          schema: v9
          index:
            prefix: index_
            period: 168h

      storage_config:
        elastic:
          address: http://es_addr

Description

I'm running a 3 node Cortex 1.4.0 cluster with -target=all and I'm seeing pretty bad query performance in Grafana. I figured my issue is not using the Query Frontend to parallelize the queries. But the documentation is quite confusing.

You can find a config of one of my nodes here.

Details

Based on the docs:

The query frontend is an optional service providing the querier’s API endpoints and can be used to accelerate the read path.

But if we check -modules we see that frontend is not optional, but rather included in the all target:

 > cortex -modules | grep frontend
query-frontend *

Which means I'm already running a query-frontend service on each node:

 > curl -s 'http://localhost:9092/services' | grep -A1 query-frontend
					<td>query-frontend</td>
					<td>Running</td>

But my query performance is very bad, so I thought that maybe I'm using the wrong endpoint. But when I checked the codebase I could not identify any special path prefix for the query-frontend: https://github.com/cortexproject/cortex/blob/23554ce028c090a4a3413ac0e35e5e1dc9fa929f/pkg/api/api.go#L414-L420 It seems to me like query-frontend is already available under the PrometheusHTTPPrefix path which is /prometheus.

I found this comment: https://github.com/cortexproject/cortex/issues/2921#issuecomment-662998729

If you're running the query-frontend in front of a Cortex cluster, the suggested way is not using the downstream URL but configuring the querier worker to connect to the query-frontend (and here we do support SRV records).

Which suggests that my configuration should have the querier talk to the query-frontend. But how is that supposed to work if I have multiple query-frontends, one for each Cortex instance? Should each Cortex instance querier have it's own query-frontend configured as frontend_worker.frontend_address?

Another thing is, why is the flag called -querier.frontend-address but the config option is frontend_worker.frontend_address?

Or should I run a separate -target=query-frontend instance of Cortex on a separate host(probably same as my Grafana) and have the querier services connect to that single query-frontend?

Description

I'm trying to use fallback_config_file with Alertmanager service but it fails with:

 msg="GET /api/prom/configs/alertmanager (500) 78.226µs Response: \"Failed to initialize the Alertmanager\\n\"

Details

I'm running 1.4.0 binary release form GitHub and I have fallback_config_file configured to point to a file I configured the same way I used to for normal Alertmanager. When I check the logs I do not see either of these two errors: https://github.com/cortexproject/cortex/blob/23554ce028c090a4a3413ac0e35e5e1dc9fa929f/pkg/alertmanager/multitenant.go#L186 https://github.com/cortexproject/cortex/blob/23554ce028c090a4a3413ac0e35e5e1dc9fa929f/pkg/alertmanager/multitenant.go#L190 But when I query the API:

curl -sv http://localhost:9101/api/prom/configs/alertmanager -H 'X-Scope-OrgID: 0'

I get back:

Failed to initialize the Alertmanager

But the code that triggers this error doesn't actually show what caused it because err is discarded: https://github.com/cortexproject/cortex/blob/23554ce028c090a4a3413ac0e35e5e1dc9fa929f/pkg/alertmanager/multitenant.go#L476-L485 So I have no clue what I'm doing wrong. The file is in place, it has read permissions for the service.

The ingester readiness endpoint fails on ingester startup if there's a unhealthy ingester within this ring. This looks to create some confusion to users (eg. https://github.com/cortexproject/cortex/issues/2913) and I'm also not much sure of this logic makes sense when running Cortex chunks storage with WAL or the Cortex blocks storage.

I'm opening this PR to have a discussion about it. In particular:

1. Why this check was introduced?
2. What would happen if we remove it?

Describe the bug Cortex can return 5xx due a single ingester failure when a tenant is being throttled (4xx). In this case, distributor can return the error from the bad ingester (5xx) even though the other 2 returned 4xx. See this.

Looking at this code seems that if we have replication factor = 2, 1 ingester down and the other 2 returning 4xx we can have for example:

4xx + 5xx + 4xx = 5xx or 5xx + 4xx + 4xx = 4xx etc

To Reproduce Steps to reproduce the behavior: I could create a unit test that reproduce the behavior: https://github.com/alanprot/cortex/commit/fd36d97e010f93e28db21e3a1e981e17cd281a80

1. Start Cortex (SHA or version) a4bf1035478641626fcbdd5fd12325c08a2bba76
2. Perform Operations(Read/Write/Others) Write Expected behavior Cortex should return the error respecting the quorum of the response from ingesters. So, if 2 ingesters return 4xx and one 5xx, cortex should return 4xx. This means that if distributor receive one 4xx and one 5xx, it needs to wait the response of the third ingester.

Environment:

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop] Kubernetes
• Deployment tool: [e.g., helm, jsonnet] Helm Storage Engine
• [X] Blocks
• [ ] Chunks

Additional Context

EDIT: Please see #703 for description

@JML - I'm not entirely sure that adding an override to Gopkg.toml was the proper fix, so if I got it wrong, let me know what the proper method is and I'll fix it up. :-)

Thanks!

Description

Every time I modify Cortex configuration and restart the nodes they generate ungodly amounts of logs like this:

msg="push error" err="rpc error: code = Code(400) desc = user=fake: sample timestamp out of order; last timestamp: ...

I assume this is because the upstream Prometheus instance is re-trying pushing of the metrics that failed when node was down.

It generates quite a lot of them...

 > sudo journalctl -a -u cortex | grep 'sample timestamp out of order' | wc -l
173476

Questions

• Is Cortex incapable of ingesting old metrics re-pushed by Prometheus? Or am I doing something wrong?
• If Cortex is incapable of ingesting old metrics, why is this an error rather than a warning or even a debug message?
• Can I stop this specific message from spamming my logs somehow?

Description

I'm running 1.4.0 using the binary from GitHub and I have ruler configured to send alerts to my own cluster of Alertmanager.

For a moment I saw the alerts in my Alertmanager Web UI, but shortly after they disappeared.

Config

My ruler section of the config looks like this:

ruler:
  external_url: 'https://alerts.example.org/'
  alertmanager_url: 'http://localhost:9093/'
  enable_alertmanager_v2: true
  rule_path: '/var/tmp/cortex/rules'
  enable_api: true
  storage:
    type: local
    local:
      directory: '/etc/cortex/rules'

My rules are located in /etc/cortex/rules/fale since I use auth_enabled: false.

Debugging

I can see the rules are located in the right place because I can look them up using the /api/v1/rules call:

 > curl -s 'http://localhost:9092/api/v1/rules' | head
instance.yml:
    - name: instance
      rules:
        - alert: InstanceDown
          expr: up == 0
          for: 5m
          annotations:
            current_value: '{{ $value }}'
            description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
            summary: Instance {{ $labels.instance }} down

But, when I try to use the /prometheus/api/v1/rules path I get nothing:

 > curl -s 'http://localhost:9092/prometheus/api/v1/rules' -H 'X-Scope-OrgID: fake' | jq .
{
  "status": "success",
  "data": {
    "groups": []
  },
  "errorType": "",
  "error": ""
}

Even though just minutes ago I saw the rules displayed here. As well as the alerts generated by the rules. But now there's nothing there:

 > curl -s 'http://localhost:9092/prometheus/api/v1/alerts' -H 'X-Scope-OrgID: fake' | jq .
{
  "status": "success",
  "data": {
    "alerts": []
  },
  "errorType": "",
  "error": ""
}

I'm confused as to what caused them to disappear. Restarting Cortex nodes doesn't fix the issue.

Questions

• My understanding is that ruler.rule_path is the place where Cortex checks for rule files. Correct?
• My understanding is that ruler.storage.local.directory configures a temporary location for rule files. Correct?
• Why can the rules be loaded from ruler.rule_path but are not available via /prometheus/api/v1/rules?

Describe the bug I'm seeing random 500s when Prometheus is pushing metrics to /api/v1/push:

msg="Failed to send batch, retrying" err="server returned HTTP status 500 Internal Server Error: rpc error: code = Unavailable desc = transport is closing"

Which looks like this on the Cortex side:

msg="POST /api/v1/push (500) 281.388644ms Response: \"rpc error: code = Unavailable desc = transport is closing\\n\" ws: false; Content-Encoding: snappy; Content-Length: 32409; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.26.0; X-Prometheus-Remote-Write-Version: 0.1.0; "

But it's just a warn level message, and even with debug logs I see no reason for this error.

The number of samples being sent is tiny:

 > curl -s localhost:9090/metrics | grep "^prometheus_tsdb_head_series "
prometheus_tsdb_head_series 34294

And the hosts are VERY beefy and underutilized, so I'm really confused why this is happening

To Reproduce Not really sure. I'm happy to help debug this, but I'm not sure where to start.

Expected behavior Error should include reason for 500 error, but all it contains is rpc error: code = Unavailable desc = transport is closing.

Environment:

• Infrastructure: Systemd service on Ubuntu
• Version: 1.8.0

Storage Engine Chunks storage using Cassandra 3.11.9.

Additional Context I started getting a LOT of 500s suddenly, so I disabled all Prometheus instances except one to debug this, but the logs give me no indication as to why it's actually happening. When I re-enable all other Prometheus instances the 500s keep raising until they overwhelm Cortex.

The ruler service in our cluster is frequently (every day) running into issues that end up meaning no rules are processed. The main issue seen is upper-percentile (90th percentile and above) ruler query time durations increase to 10 - 20 seconds, which causes the ruler to run into the group timeout (left at the default 10s in our cluster). Since we evaluate ~100 rules per tenant, these high percentile latencies cause every evaluation to fail.

Queries for this graph look like:

histogram_quantile(0.99, sum(rate(cortex_distributor_query_duration_seconds_bucket{name="ruler"}[1m])) by (le))

Lots of log messages like:

ts=2018-02-13T09:38:55.273063356Z caller=log.go:108 level=error org_id=0 msg="error in mergeQuerier.selectSamples" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
ts=2018-02-13T09:38:55.274565552Z caller=log.go:108 level=warn msg="context error" error="context deadline exceeded"

Describe the bug Cortex Compactor, upon pod redeployment, loses the current value of cortex_compactor_blocks_marked_for_no_compaction_total metric.

This might or might not be affected by the fact, that currently I'm running Cortex 1.11.0 deployed via Cortex Helm Chart 1.4.0 with all bells and whistles (caches, key stores etc) but without Cortex Compactor pod. It's deployed separately and with minimum configuration possible. It's running cortex:master-bb6b026 version in order to incorporate https://github.com/cortexproject/cortex/commit/4d751f23f8de6bc871beac595f587f12ab588388 which introduced fix to compaction process which was blocking compaction in my env.

To Reproduce Steps to reproduce the behavior:

1. Deploy Cortex
2. Start compactions
3. Wait until some blocks are marked as no compaction and the metric cortex_compactor_blocks_marked_for_no_compaction_total starts showing value >0
4. Redeploy the whole Cortex or Compactor pods only
5. The metric shows 0 now (until new no compaction block is encountered)

Expected behavior Current cortex_compactor_blocks_marked_for_no_compaction_total value is not lost upon Cortex redeployment

Environment: GKE 1.21 Cortex 1.11.0 deployed via Cortex Helm Chart 1.4.0 (without compactor pod) Cortex Compactor cortex:master-bb6b026 deployed separately

Storage Engine Blocks

Additional Context n/a

Signed-off-by: 🌲 Harry 🌊 John 🏔 [email protected]

What this PR does: Fixes an issue where the ingesters were returning empty response for metadata APIs.

Which issue(s) this PR fixes: Fixes #

Checklist

• [X] Tests updated
• [ ] Documentation added
• [X] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: sahnib [email protected]

What this PR does:

Makes TSDB max exemplars config per tenant. Note that the MaxExemplars value is passed down to prometheus tsdb at tsdb.Open call, hence this configuration would not be hot loaded at this time - unless the ingesters re-open the database handle, or go through a restart..

Which issue(s) this PR fixes: Fixes #5016

Checklist

• [X] Tests updated
• [X] Documentation added
• [X] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Is your feature request related to a problem? Please describe.

The current copy of Cortex is implemented through consistent hashing, and the location of the copy (Inageter node) is determined at the beginning of the metric data writing. In the case that some metric samples are relatively large, the load of different Ingester nodes will vary greatly.

Describe the solution you'd like Whether we can implement dynamic replicas (shards), we can dynamically schedule between different Ingester nodes based on size, time slice, load, etc.

Describe alternatives you've considered

Additional context

Is your feature request related to a problem? Please describe. Currently, Ingester regularly uploads block data to object storage, which will occupy a large amount of write bandwidth, or even fill up the bandwidth. Have we considered peak clipping for uploads? This delay should be tolerable with the current design.

Describe the solution you'd like The maximum upload flow limit can be configured to make data upload smoother.

Describe alternatives you've considered

Additional context

What this PR does: Renames oltp_endpoint to otlp_endpoint to match opentelemetry spec and lib name

Which issue(s) this PR fixes: Fixes #5067

Checklist

• [x] Tests updated
• [x] Documentation added
• [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]