What this PR does

For the record: the original PR this is based on was authored by @ryan-dyer-sp , see #2437

Replication zone support for alertmanager, ingester, store-gateway component Including migration path, tests and documentation.

The migration is written in a way so that:

1. step sets the final configuration in the Mimir YAML configuration

• this means it can be validated right at the start
• subsequent steps alter CLI options , only restart what's necessary

2. steps are using named toggles, I realized that a single number would be too hard for us to maintain.

Which issue(s) this PR fixes or relates to

Fixes #2020

Checklist

• [ ] Tests updated
• [x] Documentation added
• [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

What this PR does

Add compactor HTTP API for uploading TSDB blocks.

TODOs

• [x] Add HTTP endpoints in cloud gateway
• [x] Add HTTP endpoints in GEM gateway
• [x] Incorporate @pracucci's feedback
• [x] Add tests
• [x] Add validation when creating a block upload session
• [x] Validate block metadata
• [x] Validate block files(?) (@colega suggestion)
• [x] Make sure that when starting a backfill, file lengths are sent with file index
• [x] Validate that block time range is within retention period (@aldernero working on this)
• [x] Validate minTime/maxTime in meta.json (@aldernero working on this)
• [x] Validate block ID on backfill start
• [x] Test output from sanitizeMeta
• [x] Mark user-uploaded blocks, for debugging and security(?) (this is in place through thanos.source property in meta.json)
• [x] Make sure that a backfill can be restarted, in case it got interrupted
• [x] Add/fix tests

Which issue(s) this PR fixes or relates to

Checklist

• [x] Tests updated
• [x] Documentation added
• [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Fixes: https://github.com/grafana/mimir/issues/2266

Edit 2022-07-22: Most of the code in pkg/distributor/forwarding has been re-written so looking at the diff between the old and the new code is probably not very helpful when reviewing, I recommend looking at the new implementation as if it were completely new.

In a conversation via DMs with @bboreham we concluded that it would be better to keep the forwarding logic as isolated from the distributor's PushWithCleanup() as possible, this should help to ensure that there are no race conditions or pool usage bugs.

What this PR does: In this PR I propose to introduce a docker-compose local setup (based on single binary and memberlist) to allow the community to have a quick way to try the latest stable release of Mimir in a HA setup. It also runs Prometheus (used both to scrape Mimir metrics and run recording rules) as well as Grafana with our dashboard provisioned.

The PR includes a tutorial-style README.md guiding the user step-by-step. I've try to follow the Grafana tutorial style, but I haven't used tutorial markdown syntax given for the moment won't be published as a Grafana tutorial.

Which issue(s) this PR fixes: Fixes #991 Fixes #1024

Checklist

• [ ] Tests updated
• [ ] Documentation added
• [ ] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

What this PR does

This PR moves the active series custom trackers to a runtime configuration, and also introduces tenant-specific overwrites for these matchers. I am uploading this to start an early discussion of the design,

Key points I would already like to discuss:

• ~~Moving the custom trackers to runtime configuration is a breaking change as it removes the flag and changes the default behavior~~
• Config change management could be better handled on manager side, with some content hashing, but that would introduce dskit changes...

Which issue(s) this PR fixes

https://github.com/grafana/mimir-squad/issues/526

Fixes #

Checklist

• [x] Tests updated
• [x] Documentation added
• [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

What this PR does

Fixes this problem: currently, you can modify the docker-compose.yml directly (which is supposed to be generated) without modifying the docker-compose.jsonnet template and have it still pass the CI/CD tests, which doesn't seem to be intended, because of this issue https://stackoverflow.com/questions/3931741/why-does-make-think-the-target-is-up-to-date where if the make command name is the same as the name of a file in that folder that already exists, it does not regenerate it and hence does not notice that the file has changed. So we use .PHONY to ensure that make check in our CI/CD will actually regenerate the file properly.

Which issue(s) this PR fixes or relates to

N/A. CI/CD issue only, doesn't affect users

Checklist

• [x] Tests updated
• [x] Documentation added
• [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

This should help with building an understanding of customer requirements as far as how far back in time (wrt walclock, and wrt the sample with highest timestamp seen) samples tend to go. This whole codebase is pretty new to me, so want to check if this makes sense... Note that we don't track per-tenantID as that would be expensive and I don't think we need to. cc @codesome

Note: changes are in mimir and also in vendored prometheus. For now I made the change in vendor dir directly (hence the CI failure), but if we want to take this forward, it will be done properly.

Checklist

• [ ] Tests updated
• [ ] Documentation added
• [ ] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

What this PR does

Adds the possibility to define a fallback config for alertmanager from the helm chart config

Checklist

• [x] Tests updated
• [ ] Documentation added
• [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

In preparation of Mimir launch, we need to automate the release process:

• [x] Build binaries (+ checksums)
• [x] Build and push Docker images to docker.io
• [x] https://github.com/grafana/mimir/issues/837
• [x] https://github.com/grafana/mimir/pull/975
• [x] https://github.com/grafana/mimir/pull/1138
• [x] Push all Mimir images (including tools and build image) to docker.io/grafana/*
• [x] Revise RELEASE.md
  • [x] Release schedule: should be removed, it was from Cortex
  • [x] Change Cortex->Mimir
  • [x] Ensure release procedure is up to date
  • [x] Decide how to tag release, start at v2.0.0 -> design doc

Out of scope:

• publishing docs to the grafana.com website (tracked in another issue - cc @jdbaldry can you fill in the issue please?)
• rpm, deb, homebrew packages (in fact, get rid of any leftovers in makefile and elsewhere)
• automating upload of binaries

While investigating slow integration tests I've noticed that some of them (eg. TestSingleBinaryWithMemberlist) is significantly slowed down by assertions ring metrics, like this one:

require.NoError(t, s.Stop(mimir1))
require.NoError(t, mimir2.WaitSumMetrics(e2e.Equals(2*512), "cortex_ring_tokens_total"))

Why? Because the ring client updates the metrics every 10s (hardcoded): https://github.com/grafana/dskit/blob/84c00dae89477871dbfa0b83c823c7258f53e3bd/ring/ring.go#L288-L302

This means that every time there's a ring change (eg. s.Step(mimir1)) and then we wait for that change to be propagated (eg. mimir2.WaitSumMetrics(e2e.Equals(2*512), "cortex_ring_tokens_total")) we end up waiting up to 10s after the change has been propagated just because client metrics are not updated right after the update has been received by the ring client.

The regression has been introduced in dskit PR 50.

What this PR does

I think we should have this committed in the repo rather than having to look for it in issue comments.

Which issue(s) this PR fixes or relates to

Ref: https://github.com/grafana/mimir/issues/1537

Checklist

• [ ] Tests updated
• [x] Documentation added
• [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Describe the bug

After deploying mimir-distributed version 4.0.0

To Reproduce

Steps to reproduce the behavior:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm upgrade --install prometheus-operator prometheus-community/kube-prometheus-stack
helm fetch --untar grafana/mimir-distributed --version 4.0.0
helm upgrade --install mimir ./mimir-distributed \
  -f ./mimir-distributed/values.yaml \
  -f ./mimir-distributed/large.yaml

Expected behavior

Either be able to disable the ServiceMonitor for the gateway or add the /metrics endpoint as expected (see workaround below)

Environment

• Infrastructure: AWS EKS
• Deployment tool: Helm

Additional Context

My workaround has been to add nginx-prometheus-exporter as sidecar

gateway:
  enabledNonEnterprise: true
  extraContainers:
  - args:
    - -nginx.retries=5
    - -nginx.scrape-uri=http://127.0.0.1:8080/nginx_status
    - -web.telemetry-path=/metrics
    - -web.listen-address=:9113
    image: nginx/nginx-prometheus-exporter:0.11.0
    name: nginx-exporter
    resources:
      requests:
        cpu: 20m
        memory: 32Mi
  nginx:
    config:
      serverSnippet: |
        location = /metrics {
          proxy_pass http://127.0.0.1:9113$request_uri;
          auth_basic off;
        }
        location = /nginx_status {
          stub_status;
          auth_basic off;
        }
    verboseLogging: false
  podDisruptionBudget:
    maxUnavailable: 50%
  replicas: 5
  resources: {}

Is your feature request related to a problem? Please describe.

query-frontend caches results, but it sets a TTL of 7d to those results:

https://github.com/grafana/mimir/blob/cac269d4837e168b3b3f17b7e2e56cdacd79c4c7/pkg/frontend/querymiddleware/split_and_cache.go#L37-L38

If you're caching a year-long query, it means that you'll have to recalculate it next week again, which is undesirable.

Describe the solution you'd like

Make the TTL configurlable, allow no TTL at all (Memcache is LRU, why caring about TTL?)

Note that the maximum TTL for Memcache appears to be 30d.

What this PR does

This moves the configuration parameter reference from a sub-sub-sub page to a dedicated top-level section where it stands out more and is easier to access (imho) for someone not very familiar with the structure of the documentation.

It also adds the --init flag to the docker run command that serves docs locally. At least on my system, this was necessary to make the container shut down on Ctrl+C as advertised.

Which issue(s) this PR fixes or relates to

n/a

Checklist

• [ ] Tests updated
• [ ] Documentation added
• [ ] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Describe the bug

Helm 4.0.0 Gateway nginx config is not working with zone-aware alertmanager configuration (mimir oss)

To Reproduce

Steps to reproduce the behavior:

1. Install mimir-distributed version 3.3.x
2. Upgrade to mimir-distributed version 4.0.0 with zone-aware configuration disabled
3. Migrate to unify proxy deplouyment using following procedure https://grafana.com/docs/mimir/v2.5.x/operators-guide/deploy-grafana-mimir/migrate-to-unified-proxy-deployment/
4. Migrate alertmanager single zone to zone-aware replication using following procedure https://grafana.com/docs/mimir/latest/migration-guide/migrating-from-single-zone-with-helm/

Expected behavior

Expect the alert manager to be available through the gateway configuration but it is not.

The problem is that the gateway nginx.conf is not correct. The service "mimir-alertmanager" has been replaced by "mimir-alertmanager-zone-a", "mimir-alertmanager-zone-b" and "mimir-alertmanager-zone-c"

    # Alertmanager endpoints

    location /alertmanager {

      proxy_pass      http://mimir-alertmanager.mimir.svc.cluster.local:8080$request_uri;

    }

    location = /multitenant_alertmanager/status {

      proxy_pass      http://mimir-alertmanager.mimir.svc.cluster.local:8080$request_uri;

    }

    location = /api/v1/alerts {

      proxy_pass      http://mimir-alertmanager.mimir.svc.cluster.local:8080$request_uri;

    }

resolution proposal

change the gateway nginx.conf (see: https://github.com/grafana/mimir/blob/mimir-distributed-4.0.0/operations/helm/charts/mimir-distributed/values.yaml#L2485) to have something like that

    # Alertmanager endpoints

    location /alertmanager {

      proxy_pass      http://mimir-alertmanager-headless.mimir.svc.cluster.local:8080$request_uri;

    }

    location = /multitenant_alertmanager/status {

      proxy_pass      http://mimir-alertmanager-headless.mimir.svc.cluster.local:8080$request_uri;

    }

    location = /api/v1/alerts {

      proxy_pass      http://mimir-alertmanager-headless.mimir.svc.cluster.local:8080$request_uri;

    }

What this PR does

This adds an alert to detect alertmanager instances that don't own any tenants.

Which issue(s) this PR fixes or relates to

Fixes #1959

Checklist

• [ ] Tests updated
• [x] Documentation added
• [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

What this PR does

This introduces the querier.max-partial-query-length flag to allow limiting the time range for (partial) queries at the querier level. It also deprecates store.max-query-length which became ambiguous due to its different semantics in limiting query ranges at the frontend and querier levels.

Which issue(s) this PR fixes or relates to

Fixes #2793

Checklist

• [x] Tests updated
• [ ] Documentation added
• [x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]