Hi!

In new version of alertmanager API version will be changed to v2. URI /api/v1/ will no longer work. And dashboard not work - request to http://alertmanager:9093/api/v1/alerts/groups failed with 404 Not Found

docker image for test prometheus/alertmanager:master

I use Karma with alertmanager.proxy: true and authentication/authorization handled by a reverse proxy (Cloudflare Access to be precise).

The issue is that the session tokens set by Cloudflare Access expire after a while, so when Karma tries to fetch from /alerts.json, it gets an HTTP 302 reply and stops there. If it followed the redirects instead, it would be able to renew its token (through the magic of SSO), and then it could fetch the alerts as if nothing had happened.

Does that make sense?

We use Karma on top of an AlertManager high availability cluster (with 2 or more AM). We use filter on labels. Sometimes no alerts rendering on Karma. In this example, the number on top indicate that no alert match 'pf="sdr"' but in AM we can show the alerts.

We don't see error in Karma logs

time="2019-07-09T07:06:46Z" level=info msg="Pulling latest alerts and silences from Alertmanager"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Collecting alerts and silences"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Collecting alerts and silences"
time="2019-07-09T07:06:46Z" level=info msg="GET http://alertmanager.xymon:9093/metrics timeout=10s"
time="2019-07-09T07:06:46Z" level=info msg="GET http://alertmanager.prometheus:9093/metrics timeout=10s"
time="2019-07-09T07:06:46Z" level=info msg="Upstream version: 0.18.0"
time="2019-07-09T07:06:46Z" level=info msg="GET http://alertmanager.prometheus:9093/api/v1/status timeout=10s"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Remote Alertmanager version: 0.18.0"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Got 3 silences(s) in 764.867µs"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Detecting JIRA links in silences (3)"
time="2019-07-09T07:06:46Z" level=info msg="Upstream version: 0.17.0"
time="2019-07-09T07:06:46Z" level=info msg="GET http://alertmanager.xymon:9093/api/v1/status timeout=10s"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Remote Alertmanager version: 0.17.0"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Got 0 silences(s) in 4.333911ms"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Detecting JIRA links in silences (0)"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Got 136 alert group(s) in 67.698158ms"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Deduplicating alert groups (136)"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Processing unique alert groups (56)"
time="2019-07-09T07:06:46Z" level=info msg="[sdr] Merging autocomplete data (480)"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Got 997 alert group(s) in 585.932343ms"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Deduplicating alert groups (997)"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Processing unique alert groups (838)"
time="2019-07-09T07:06:46Z" level=info msg="[xymon] Merging autocomplete data (2908)"
time="2019-07-09T07:06:46Z" level=info msg="Pull completed"

time="2019-07-09T06:59:34Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 15.087184ms"
time="2019-07-09T07:00:34Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 18.785712ms"
time="2019-07-09T07:01:35Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 15.025122ms"
time="2019-07-09T07:02:35Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 11.880314ms"
time="2019-07-09T07:03:36Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 16.137345ms"
time="2019-07-09T07:04:37Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 19.860792ms"
time="2019-07-09T07:05:37Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 18.43056ms"
time="2019-07-09T07:06:38Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 21.778087ms"
time="2019-07-09T07:07:38Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 22.852808ms"
time="2019-07-09T07:08:39Z" level=info msg="[10.244.6.0 MIS] <200> GET /alerts.json?q=pf%3Dsdr&q=%40state%3Dactive took 14.763488ms"

Did you encounter this problem ?

We authenticate to karma using apache

But now somehow with the rewrite to react, the /alerts.json is not authenticated like it was with unsee. In the console I see 401 errors on that endpoint.

Hi. I've got strange situation. I see the alert in Karma with non-legal receiver I expect to see team-kafka-wake-up there

{
          "annotations": {
            "summary": "K2DWH Lag is growing. Info: group=k2dwh, count=8.172123e+06, location=fr"
          },
          "endsAt": "2020-06-11T09:54:41.353Z",
          "fingerprint": "52034abab298fe32",
          "receivers": [
            {
              "name": "team-kafka-wake-up"
            },
            {
              "name": "team-kafka-wake-up"
            }
          ],
          "startsAt": "2020-06-11T09:45:26.353Z",
          "status": {
            "inhibitedBy": [],
            "silencedBy": [],
            "state": "active"
          },
          "updatedAt": "2020-06-11T09:51:41.451Z",
          "generatorURL": "https://****",
          "labels": {
            "alertname": "k2dwh_lag",
            "consumer_group": "k2dwh",
            "kafka_location": "fr",
            "severity": "critical",
            "team": "Team_Kafka"
          }
        },

See video on https://www.dropbox.com/s/lfyr7jv8n6dwzur/Screen%20Recording%202019-05-22%20at%2010.34.26.mov?dl=0

Running with chart v1.1.13 which is karma 0.34: https://github.com/helm/charts/blob/master/stable/karma/values.yaml#L9

First of all, thank you for developing this dashboard! It is very handy for us at trivago.

Would it be possible to have a "Nightmode" or "Dark mode" setting in the dashboard?

Some of us work in a low luminosity environment and the karma dashboard is bright. It strains the eye after a while.

If the little alarm windows could have a dark gray background for example, and the window could have an even darker shade of gray in this option, that would be great! Grafana does this very well.

Do you think this would be possible? It would certainly make some of your users very happy!

in Karma V109 we are seeing some weird behaviour with the summary annotations in groups. We have the following set so the summary annotation should always be visible:

    annotations:
      default:
        hidden: false
      visible:
        - summary
      order:
        - summary

but weve noticed that on grouped alerts, if the summary annotation is unique it shows up, but if its the same for all group members it moves to the bottom of the card and you can only see it by clicking the plus. This is unexpected behaviour as we have the app configured to always show summary annotation.

example: There is no summary annotation displayed.

but if we click on the plus: The summary annotation appears.

Hi,

We upgraded to 0.95 this morning, and following on from https://github.com/prymitive/karma/pull/3881, we're now seeing silences not matching correctly when special chars are escaped in the regex.

Example - Top matcher is an expired alert, and bottom is a new remaking of it that's failing to match:

Hi, I have karma deployed behind a proxy from Rancher. So the karma URL is something like https://example.com/k8s/clusters/project/api/v1/namespaces/monitoring/services/http:karma:80/proxy/?q=

When the proxy mode is enabled, silence management request will be sent to https://example.com/proxy/alertmanager, instead of https://example.com/k8s/clusters/project/api/v1/namespaces/monitoring/services/http:karma:80/proxy/alertmanager

Both list alerts and list silence work find, but silence management requests are sent to an invalid URL.

I found the code that returning InternalURI wrote: https://github.com/prymitive/karma/blob/main/internal/alertmanager/models.go#L183-L195

// InternalURI is the URI of this Alertmanager that will be used for all request made by the UI
func (am *Alertmanager) InternalURI() string {
	if am.ProxyRequests {
		sub := fmt.Sprintf("/proxy/alertmanager/%s", am.Name)
		if strings.HasPrefix(config.Config.Listen.Prefix, "/") {
			return path.Join(config.Config.Listen.Prefix, sub)
		}
		return path.Join("/"+config.Config.Listen.Prefix, sub)
	}

	// strip all user/pass information, fetch() doesn't support it anyway
	return uri.WithoutUserinfo(am.PublicURI())
}

It would be nice if InternalURI can be customized. And if no customized URL is specified, fallback to this code, for backward-compatible

Hello,

Thanks for the awesome project -- we use it everyday to help us monitor our network.

We have this running on a large TV and we want an easy way to identify critical alerts from far away. The colorTitlebar feature is nice, but is limited to red (non-silenced) or green (silenced).

It would be great if I could pass in a custom mapping of severity to color for the titlebar (similar to how the label color config is done). When using this feature, you could force the alert count to have a black background with white text.

The yaml could maybe look something like this (just a sketch, feel free to change):

titlebar:
  color:
    silenced: "#18bc9c"
    custom:
      severity:
        - value: info
          color: "#87c4e0"
        - value: warning
          color: "#ffae42"
        - value: critical
          color: "#ff220c"

Thanks

The default Alertmanager<->OpsGenie integration is using 'annotation.priority' as it's preferred field to determine severity-level. I would like to show these alerts in Karma and match the priority-color according to OpsGenie colors. However, the configuration of Karma seems to only allow 'labels' as colorable.

I have 2 basic ideas:

1. Expand the config to allow of colorizing 'annotations'
2. Add a setting/function to 'promote' certain annotations into labels (thus using the existing config) (this could also allow a 'demote'-setting to convert a label into an annotation)

I'm willing to try to do a PR for option 2, but only if you feel it's a good-enough option

Is possible to scrape alertmanger targets using consul discovery ?

Prometheus example:

scrape_configs:
  - job_name: 'haproxy'
    consul_sd_configs:
    - server: 'localhost:8500'
      services:
        - 'haproxy_exporter'

I think this can be useful to add alertmanager instances running in aws ec2 or k8s.

We are using victoria-metrics with http basic auth as the database that stores the history so i implemented this feature. Im not a go programmer so something may not be written in the best way, sorry for that.