Would help if bosun supported influxdb. I didn't find a bug tracking this, so here it is.

Since I have multiple datasources sending data to influxdb. (collectd, statsite). It would keep my dependencies low if I could use the influxdb for bosun rather than change the entire system to openTSDB.

Is it possible to have multiple instances of the same type of backends? for example multiple InfluxDB backends or multiple ElasticSearch backends? I ask because I'm trying to pull in data from two separate instances but simply creating a duplicate key results in a config error: fatal: main.go:88: conf: bosun.config:2:0: at <influxHost = xx.xx.x...>: duplicate key: influxHost

This is a solution for #2065

The idea behind this is simple. Every check run is slightly shifted so that the checks are distributed uniformly.

For the subset of checks that run with the period T, a shift is added to every check. The shift ranges from 0 to T-1. The shifts are incremental. For example, if we have 6 checks every 5 mins (T=5). The shifts will be 0, 1, 2, 3, 4, 0. This way, without the patch 6 checks will happen at times 0, and 5; with the patch, two checks will happen at the time 0, one at 1, one at 2, and so on. The total number of checks and check period stay the same.

Here is the test that shows the effect of the patch on system load. Note, that the majority of checks in this system have 5 mins period.

I want to deploy bosun as a dashboard & alerting system within my organization, but I feel like having config management being completely external to bosun is a major drawback. It would be super fantastic if it were possible to, entirely through the web interface, define, test, and commit a new alert, or to update an existing alert to tweak the parameters.

Is anything like this in the works? How do you manage this in your existing deployments?

Problem: Something goes down which results in lots of other things being down, because of this, we get a lot of alerts.

Common Examples:

• A Network Partition: Some portion of hosts become unavailable from bosun's perspective
• Host Goes Down: Everything monitored on that host becomes unavailable
• Service dependencies: We expect some service to go down if another service goes down
• Bosun can't query it's database (This is probably a different feature, but noting here nonetheless)

Things I want to be able to do based on our config at Stack Exchange:

• Have our host-based alert macro include detect if the host in Oregon (because the host name has "or-". So this is basically a dependency based on a lookup table
• Have our host-based alerts not trigger if bosun is unable to ping the host (which would be another alert most likely)
• Be able to have dependencies for alerts that may have no group.

The status for any alert that is not triggering for an alert should be "unevaluated". This won't show up on the dashboard or trigger notifications.

Two general approaches come to mind. The first is that dependencies require another alert. That other alert is run first, and the alert won't trigger based on the result of another alert. The other is that dependencies are an expression. I think the expression route only really makes sense if an alert itself can be used as an expression.

Another possibility which I haven't thought much about is that alerts generate dependencies and not the other way around. So for example, an alert marks some tagset as something that should not be evaluated.

Making Stuff Up....

macro ping_location {
    template = ping.location
    $pq = max(q("sum:bosun.ping.timeout{dst_host=$loc*,host=$source}", "5m", ""))
    $grouped = t($pq,"")
    $hosts_timing_out = sum($grouped)
    $total_hosts = len($grouped)
    $percent_timeout = $hosts_timing_out / $total_hosts * 100
    crit = $percent_timeout > 10
}

#group is empty
alert or_hosts_down {
    $source=ny-bosun01
    $loc = or-
    $name = OR Peak
    macro = ping_location
}

#Group is {dst_host=*}
alert host_down {
   template = host_down
   cirt = max(q("sum:bosun.ping.timeout{dst_host=*", "5m", ""))
}

lookup location {
    entry host=or-* {
        alert = alert("or_hosts_down")
    }
    ...
}

macro host_based {
   #This makes it so alerts based on this macro that are host based won't trigger if 
   dependency = lookup("location", "alert") || alert("host_down")
   #Another idea here is that you can create tag synonyms for an alert. So instead of having to add this lookup function that translates, have a synonym feature of alerts and also global that says (consider this tag key to be the same as this tag key). This would also solve an issue with silences (i.e. silencing host=ny-web11 doesn't do anything for the haproxy alert that has hosts as svname). Another issue with that is the those alerts are not tag based, so we actually need inhibit in that case. 
}

We have a very simple rule file, with 3 notifications (http post to PD and slack, and email) and a bunch of alert rules which trigger them. We are facing a weird issue wherein, the following happens:

• alert triggers, sends notifications
• a human acks the alert
• human solves problem, alert becomes inactive
• human closes the alert
• notification still keeps triggering (alert is no where to be seen in the bosun UI/api) - forever!

to explain it through logs, quite literally this is what we're seeing:

2016/04/01 07:56:37 info: check.go:513: check alert masked.masked.write.rate.too.low start 2016/04/01 07:26:38 info: check.go:537: check alert masked.masked.write.rate.too.low done (1.378029647s): 0 crits, 0 warns, 0 unevaluated, 0 unknown 2016/04/01 07:26:38 info: alertRunner.go:55: runHistory on masked.masked.write.rate.too.low took 54.852815ms 2016/04/01 07:26:39 info: search.go:205: Backing up last data to redis 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka} 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka} 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka} 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka} 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka} 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka} 2016/04/01 07:28:20 info: notify.go:115: relayed alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka} to [[email protected]] sucessfully. Subject: 148 bytes. Body: 3500 bytes. 2016/04/01 07:28:20 info: notify.go:115: relayed alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka} to [[email protected]] sucessfully. Subject: 147 bytes. Body: 3497 bytes. 2016/04/01 07:28:20 info: notify.go:80: post notification successful for alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka}. Response code 200. 2016/04/01 07:28:20 info: notify.go:80: post notification successful for alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka}. Response code 200. 2016/04/01 07:28:20 info: notify.go:80: post notification successful for alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka}. Response code 200. 2016/04/01 07:28:20 info: notify.go:80: post notification successful for alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka}. Response code 200.

Hi all, as described in the docs, I'm using the templates subject as body for POSTing stuff to our hipchat bot. the problem I encounter is in Bosun main view (list of alerts) where the template subject is presented when clicking an alert for details.

Suggested is to use templates' body as payload for notification (POST notifications mainly). a flag can be also added to let the user which template will use the subject as payload and which will use the body.

Thanks, Yarden

When an alert instance goes from (Unknown Warning or Critical) to Normal a recovery email should be sent.

Considerations:

• Should recovery templates be their own template? I think they should, and repeated logic can be done via include templates
  • Who to notify? The same notifications that were notified of the previous state.
  • notifications will need a no_recovery option. This is needed if we want to hook up alerts to pagerduty (don't want our phones being dialed to let us know that an issue is recovered, at that point we can rely on email)

My main reservation about this feature is that users are more likely not to investigate an alert that is recovered, this is dangerous because the alert could be a latent issue. However, it is better to provide a better frictionless workflow than a road block. Bosun aims to provide all the tools needed for very informative notifications so good judgements can be made at times without needing to go to a console. Furthermore, we should also add acknowledgement notifications. This will be a way to inform all recipients of an alert that someone has made a decision about this alert and hopefully committed to an action (fixing the actual problem, or tuning the alert).

Ack emails will be described in another issue.

This feature needs discussion and review prior to implementation.

I updated our test servers to the latest version of bosun from https://github.com/bosun-monitor/bosun/releases/download/20150428222252/bosun-linux-amd64 After running for slightly less than a day, it stopped responding.

The command line where I started it revealed:

 ./bosun-linux-amd64 -c=/data/bosun.conf
2015/05/04 16:21:54 enabling syslog
Killed

Syslog (cat /var/log/messages |grep bosun) did not reveal any log messages in the hours before the crash.

It looks like a memory leak. The graph of bosun.collect.alloc grew gradually from 200Mb after deploying the new version to 12Gb just before the "crash":

Looking back over the last week at the memory behaviour of the previous version, there was a similar memory growth pattern in the previous version too but at a much slower rate. The bottom graph shows gradual memory increasing over the course of a week followed by two rapid increases for the newer version.

Just for interest sake, here is a general Bosun dashboard; the other stats look reasonable. Although there is a high number of go routines after restarting Bosun this appears unrelated to the leak.

More information about our setup:

• Backend: OpenTSDB
• Data is being passed through Bosun to OpenTSDB (as visible from the dashboard)
• We send data points every minute at a rate of about 37000 per minute
• In addition scollector is submitting data from one machine monitoring openTSDB, elasticsearch, Bosun, Linux and os
• The rule file is still a small prototype:

httpListen = :8070
tsdbHost = localhost:4242

smtpHost = ******
emailFrom = ******

macro grafanaConfig {
    $grafanaHost = ******
}

notification emailIzak {
    email = [email protected]
    next = emailIzak
    timeout = 24h
}


##################### Templates #######################


template generic {
    body = `{{template "genericHeader" .}}
    {{template "genericDef" .}}

    {{template "genericTags" .}}

    {{template "genericComputation" .}}

     {{if .Alert.Vars.graph}}
     <h3>{{.Alert.Vars.graphTitle}}</h3>
    <p>{{.Graph .Alert.Vars.graph}}
    {{end}}`

    subject =  {{.Last.Status}}: {{.Alert.Name}} on instance {{.Group.serviceinstance}}
}

template genericHeader {   
    body = `
    <h3> Possible actions </h3>   
    {{if .Alert.Vars.note}}
        <p>{{.Alert.Vars.note}}
    {{end}}
     <p><a href="{{.Ack}}">Acknowledge alert</a>

    {{if .Alert.Vars.grafanaDash}}
        <p><a href="{{.Alert.Vars.grafanaDash}}"> View the relevant statistics dasboard </a>
    {{end}}
    `
}

template genericDef {
    body = `
    <h3> Details </h3>
    <p><strong>Alert definition:</strong>
    <table>
        <tr>
            <td>Name:</td>
            <td>{{replace .Alert.Name "." " " -1}}</td></tr>
        <tr>
            <td>Warn:</td>
            <td>{{.Alert.Warn}}</td></tr>
        <tr>
            <td>Crit:</td>
            <td>{{.Alert.Crit}}</td></tr>
    </table>`
}

template genericTags {
    body = `<p><strong>Tags</strong>

    <table>
        {{range $k, $v := .Group}}
            {{if eq $k "host"}}
                <tr><td>{{$k}}</td><td><a href="{{$.HostView $v}}">{{$v}}</a></td></tr>
            {{else}}
                <tr><td>{{$k}}</td><td>{{$v}}</td></tr>
            {{end}}
        {{end}}
    </table>`
}

template genericComputation {
    body = `
    <p><strong>Computation</strong>

    <table>
        {{range .Computations}}
            <tr><td><a href="{{$.Expr .Text}}">{{.Text}}</a></td><td>{{.Value}}</td></tr>
        {{end}}
    </table>`
}

template unkown {
    subject = {{.Name}}: {{.Group | len}} unknown alerts. 
    body = `
    <p>Unknown alerts imply no data is being recorded for their monitored time series. Therefore we cannot know what is happening. 
    <p>Time: {{.Time}}
    <p>Name: {{.Name}}
    <p>Alerts:
    {{range .Group}}
        <br>{{.}}
    {{end}}`
}

unknownTemplate = unkown


#################### alerts #######################


alert FlowRouterBytesZero {
    template = generic
    $query = "sum:bytes.bytes.counter.value{serviceinstance=*}"

    $note = The flow router has reported zero bytes in the last 2 minutes. This note should contain extra information specifying what action the operator should take to resolve it. 
    $graph =q($query, "24h", "")
    $graphTitle = Flow router traffic in the last 24 hours
    macro = grafanaConfig
    $grafanaDash = $grafanaHost/dashboard/db/per-flow-route-bytes-drill-down

    $avgBytesPer2Min = avg(q($query, "2m", ""))
    $avgBytesPer5Min = avg(q($query, "5m", ""))

    warn =  $avgBytesPer2Min == 0
    crit =  $avgBytesPer5Min == 0
    critNotification = emailIzak
}

This PR adds an aggregate DSL function, which allows one to combine different series in a seriesSet using a specified aggregator (currently min, max, p50, avg).

This is particularly useful when comparing data across different weeks (using the over) function. In our case, for anomaly detection, we want to compare the current day's data with an aggregated view of the same day in previous weeks. In particular, we want to compare each point in the last day to the median of each point in the corresponding day for the last 3 weeks, so that any anomalies that occurred in a previous week are ignored. This way we compare with a hypothetical "perfect" day.

For example:

$weeks = over("avg:10m-avg-zero:os.cpu", "24h", "1w", 3)
$a = aggregate($weeks, "", "p50")
merge($a, $q)

Which looks like this:

Or, if we wanted to combine series but maintain the region and color groups`, that query would look like this:

$weeks = over("avg:10m-avg-zero:os.cpu{region=*,color=*}", "24h", "1w", 3)
aggregate($weeks, "region,color", "p50")

which would result in one merged series for each unique region/color combination.

I am very happy to take suggestions for changes / improvements. With regards to naming the function, I would have probably chosen "merge", but since that is already taken, I went with the OpenTSDB terminology and used "aggregate".

I have installed Hbase, opentsdb and bosun on a machine running Centos7. I can see the bosun website fine, but any query I try to run from the graph page is giving some error. I've put the bosun output into a log file, and there are 2 kinds of errors that pop up. Sometime it's too many open files:

2016/03/04 11:10:23 error: queue.go:102: Post http://localhost:8070/api/put: dial tcp 127.0.0.1:8070: socket: too many open files

Sometimes it's just a timeout.

2016/03/04 11:14:06 error: queue.go:102: Post http://localhost:8070/api/put: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Sometimes restarting seems to help, other times not so much. The longest I've had bosun running without these errors is a day.

Bumps github.com/aws/aws-sdk-go from 1.31.12 to 1.33.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

https://github.com/bosun-monitor/bosun/issues/2505

Description

Fixes #2505 (fill in)

Type of change

• [x] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update

How has this been tested?

• [ ] Test A
• [ ] Test B

Checklist:

• [x] This contribution follows the project's code of conduct
• [x] This contribution follows the project's contributing guidelines
• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [ ] Any dependent changes have been merged and published in downstream modules

Description

Scollector did not manage to collect data from HAProxy (HAProxy version 2.0.13-2ubuntu0.5). Got error:

Apr 28 16:26:34 ServerName scollector[1741859]: error: interval.go:65: haproxy-1-http://localhost:1936/;csv: unknown check status * L4TOUT
Apr 28 16:26:49 ServerName scollector[1741859]: error: interval.go:65: haproxy-1-http://localhost:1936/;csv: unknown check status * L4TOUT

Print from HAProxy:

Simply added "* L4TOUT" so that its a valid check status for haproxyCheckStatus

Type of change

• [x] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update

How has this been tested?

• [x] HAProxy collection now works again for HAProxy version 2.0.13-2ubuntu0.5

Checklist:

• [x] This contribution follows the project's code of conduct
• [x] This contribution follows the project's contributing guidelines
• [ ] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] New and existing unit tests pass locally with my changes
• [ ] Any dependent changes have been merged and published in downstream modules

We package this for NixOS, and we like to use the latest stable release from upstream.

https://github.com/bosun-monitor/bosun/releases/tag/0.8.0-preview is listed as the latest release on GitHub. Is it a stable release or should it be marked pre-release? I ask because it has the "-preview" suffix attached to it, making me think it is an unstable release.