⛑ Gatus - Automated service health dashboard

Last update: Dec 31, 2022

Comments: 17

A service health dashboard in Go that is meant to be used as a docker image with a custom configuration file.

I personally deploy it in my Kubernetes cluster and let it monitor the status of my core applications: https://status.twinnation.org/

Features
Usage
Docker
Running the tests
Using in Production
FAQ

Features

The main features of Gatus are:

Highly flexible health check conditions: While checking the response status may be enough for some use cases, Gatus goes much further and allows you to add conditions on the response time, the response body and even the IP address.
Ability to use Gatus for user acceptance tests: Thanks to the point above, you can leverage this application to create automated user acceptance tests.
Very easy to configure: Not only is the configuration designed to be as readable as possible, it's also extremely easy to add a new service or a new endpoint to monitor.
Alerting: While having a pretty visual dashboard is useful to keep track of the state of your application(s), you probably don't want to stare at it all day. Thus, notifications via Slack, Mattermost, Messagebird, PagerDuty and Twilio are supported out of the box with the ability to configure a custom alerting provider for any needs you might have, whether it be a different provider or a custom application that manages automated rollbacks.
Metrics
Low resource consumption: As with most Go applications, the resource footprint that this application requires is negligibly small.
GitHub uptime badges:
Service auto discovery in Kubernetes (ALPHA)

Usage

By default, the configuration file is expected to be at config/config.yaml.

You can specify a custom path by setting the GATUS_CONFIG_FILE environment variable.

Here's a simple example:

metrics: true         # Whether to expose metrics at /metrics
services:
  - name: twinnation  # Name of your service, can be anything
    url: "https://twinnation.org/health"
    interval: 30s     # Duration to wait between every status check (default: 60s)
    conditions:
      - "[STATUS] == 200"         # Status must be 200
      - "[BODY].status == UP"     # The json path "$.status" must be equal to UP
      - "[RESPONSE_TIME] < 300"   # Response time must be under 300ms
  - name: example
    url: "https://example.org/"
    interval: 30s
    conditions:
      - "[STATUS] == 200"

This example would look like this:

Note that you can also add environment variables in the configuration file (i.e. $DOMAIN, ${DOMAIN})

Configuration

Parameter	Description	Default
`debug`	Whether to enable debug logs	`false`
`metrics`	Whether to expose metrics at /metrics	`false`
`storage`	Storage configuration	`{}`
`storage.file`	File to persist the data in. If not set, storage is in-memory only.	`""`
`services`	List of services to monitor	Required `[]`
`services[].name`	Name of the service. Can be anything.	Required `""`
`services[].group`	Group name. Used to group multiple services together on the dashboard. See Service groups.	`""`
`services[].url`	URL to send the request to	Required `""`
`services[].method`	Request method	`GET`
`services[].insecure`	Whether to skip verifying the server's certificate chain and host name	`false`
`services[].conditions`	Conditions used to determine the health of the service	`[]`
`services[].interval`	Duration to wait between every status check	`60s`
`services[].graphql`	Whether to wrap the body in a query param (`{"query":"$body"}`)	`false`
`services[].body`	Request body	`""`
`services[].headers`	Request headers	`{}`
`services[].dns`	Configuration for a service of type DNS. See Monitoring using DNS queries	`""`
`services[].dns.query-type`	Query type for DNS service	`""`
`services[].dns.query-name`	Query name for DNS service	`""`
`services[].alerts[].type`	Type of alert. Valid types: `slack`, `pagerduty`, `twilio`, `mattermost`, `messagebird`, `custom`	Required `""`
`services[].alerts[].enabled`	Whether to enable the alert	`false`
`services[].alerts[].failure-threshold`	Number of failures in a row needed before triggering the alert	`3`
`services[].alerts[].success-threshold`	Number of successes in a row before an ongoing incident is marked as resolved	`2`
`services[].alerts[].send-on-resolved`	Whether to send a notification once a triggered alert is marked as resolved	`false`
`services[].alerts[].description`	Description of the alert. Will be included in the alert sent	`""`
`alerting`	Configuration for alerting	`{}`
`alerting.slack`	Configuration for alerts of type `slack`	`{}`
`alerting.slack.webhook-url`	Slack Webhook URL	Required `""`
`alerting.pagerduty`	Configuration for alerts of type `pagerduty`	`{}`
`alerting.pagerduty.integration-key`	PagerDuty Events API v2 integration key.	Required `""`
`alerting.twilio`	Settings for alerts of type `twilio`	`{}`
`alerting.twilio.sid`	Twilio account SID	Required `""`
`alerting.twilio.token`	Twilio auth token	Required `""`
`alerting.twilio.from`	Number to send Twilio alerts from	Required `""`
`alerting.twilio.to`	Number to send twilio alerts to	Required `""`
`alerting.mattermost`	Configuration for alerts of type `mattermost`	`{}`
`alerting.mattermost.webhook-url`	Mattermost Webhook URL	Required `""`
`alerting.mattermost.insecure`	Whether to skip verifying the server's certificate chain and host name	`false`
`alerting.messagebird`	Settings for alerts of type `messagebird`	`{}`
`alerting.messagebird.access-key`	Messagebird access key	Required `""`
`alerting.messagebird.originator`	The sender of the message	Required `""`
`alerting.messagebird.recipients`	The recipients of the message	Required `""`
`alerting.custom`	Configuration for custom actions on failure or alerts	`{}`
`alerting.custom.url`	Custom alerting request url	Required `""`
`alerting.custom.method`	Request method	`GET`
`alerting.custom.insecure`	Whether to skip verifying the server's certificate chain and host name	`false`
`alerting.custom.body`	Custom alerting request body.	`""`
`alerting.custom.headers`	Custom alerting request headers	`{}`
`security`	Security configuration	`{}`
`security.basic`	Basic authentication security configuration	`{}`
`security.basic.username`	Username for Basic authentication	Required `""`
`security.basic.password-sha512`	Password's SHA512 hash for Basic authentication	Required `""`
`disable-monitoring-lock`	Whether to disable the monitoring lock	`false`
`web`	Web configuration	`{}`
`web.address`	Address to listen on	`0.0.0.0`
`web.port`	Port to listen on	`8080`

For Kubernetes configuration, see Kubernetes

Conditions

Here are some examples of conditions you can use:

Condition	Description	Passing values	Failing values
`[STATUS] == 200`	Status must be equal to 200	200	201, 404, ...
`[STATUS] < 300`	Status must lower than 300	200, 201, 299	301, 302, ...
`[STATUS] <= 299`	Status must be less than or equal to 299	200, 201, 299	301, 302, ...
`[STATUS] > 400`	Status must be greater than 400	401, 402, 403, 404	400, 200, ...
`[STATUS] == any(200, 429)`	Status must be either 200 or 420	200, 429	201, 400, ...
`[CONNECTED] == true`	Connection to host must've been successful	true, false
`[RESPONSE_TIME] < 500`	Response time must be below 500ms	100ms, 200ms, 300ms	500ms, 501ms
`[IP] == 127.0.0.1`	Target IP must be 127.0.0.1	127.0.0.1	0.0.0.0
`[BODY] == 1`	The body must be equal to 1	1	`{}`, `2`, ...
`[BODY].user.name == john`	JSONPath value of `$.user.name` is equal to `john`	`{"user":{"name":"john"}}`
`[BODY].data[0].id == 1`	JSONPath value of `$.data[0].id` is equal to 1	`{"data":[{"id":1}]}`
`[BODY].age == [BODY].id`	JSONPath value of `$.age` is equal JSONPath `$.id`	`{"age":1,"id":1}`
`len([BODY].data) < 5`	Array at JSONPath `$.data` has less than 5 elements	`{"data":[{"id":1}]}`
`len([BODY].name) == 8`	String at JSONPath `$.name` has a length of 8	`{"name":"john.doe"}`	`{"name":"bob"}`
`[BODY].name == pat(john*)`	String at JSONPath `$.name` matches pattern `john*`	`{"name":"john.doe"}`	`{"name":"bob"}`
`[BODY].id == any(1, 2)`	Value at JSONPath `$.id` is equal to `1` or `2`	1, 2	3, 4, 5
`[CERTIFICATE_EXPIRATION] > 48h`	Certificate expiration is more than 48h away	49h, 50h, 123h	1h, 24h, ...

Placeholders

Placeholder	Description	Example of resolved value
`[STATUS]`	Resolves into the HTTP status of the request	404
`[RESPONSE_TIME]`	Resolves into the response time the request took, in ms	10
`[IP]`	Resolves into the IP of the target host	192.168.0.232
`[BODY]`	Resolves into the response body. Supports JSONPath.	`{"name":"john.doe"}`
`[CONNECTED]`	Resolves into whether a connection could be established	`true`
`[CERTIFICATE_EXPIRATION]`	Resolves into the duration before certificate expiration	`24h`, `48h`, 0 (if not using HTTPS)
`[DNS_RCODE]`	Resolves into the DNS status of the response	NOERROR

Functions

Function	Description	Example
`len`	Returns the length of the object/slice. Works only with the `[BODY]` placeholder.	`len([BODY].username) > 8`
`pat`	Specifies that the string passed as parameter should be evaluated as a pattern. Works only with `==` and `!=`.	`[IP] == pat(192.168.*)`
`any`	Specifies that any one of the values passed as parameters is a valid value. Works only with `==` and `!=`.	`[BODY].ip == any(127.0.0.1, ::1)`

NOTE: Use pat only when you need to. [STATUS] == pat(2*) is a lot more expensive than [STATUS] < 300.

Alerting

Configuring Slack alerts

alerting:
  slack: 
    webhook-url: "https://hooks.slack.com/services/**********/**********/**********"
services:
  - name: twinnation
    url: "https://twinnation.org/health"
    interval: 30s
    alerts:
      - type: slack
        enabled: true
        description: "healthcheck failed 3 times in a row"
        send-on-resolved: true
      - type: slack
        enabled: true
        failure-threshold: 5
        description: "healthcheck failed 5 times in a row"
        send-on-resolved: true
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"

Here's an example of what the notifications look like:

Configuring PagerDuty alerts

It is highly recommended to set services[].alerts[].send-on-resolved to true for alerts of type pagerduty, because unlike other alerts, the operation resulting from setting said parameter to true will not create another incident, but mark the incident as resolved on PagerDuty instead.

alerting:
  pagerduty: 
    integration-key: "********************************"
services:
  - name: twinnation
    url: "https://twinnation.org/health"
    interval: 30s
    alerts:
      - type: pagerduty
        enabled: true
        failure-threshold: 3
        success-threshold: 5
        send-on-resolved: true
        description: "healthcheck failed 3 times in a row"
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"

Configuring Twilio alerts

alerting:
  twilio:
    sid: "..."
    token: "..."
    from: "+1-234-567-8901"
    to: "+1-234-567-8901"
services:
  - name: twinnation
    interval: 30s
    url: "https://twinnation.org/health"
    alerts:
      - type: twilio
        enabled: true
        failure-threshold: 5
        send-on-resolved: true
        description: "healthcheck failed 5 times in a row"
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"

Configuring Mattermost alerts

alerting:
  mattermost: 
    webhook-url: "http://**********/hooks/**********"
    insecure: true
services:
  - name: twinnation
    url: "https://twinnation.org/health"
    interval: 30s
    alerts:
      - type: mattermost
        enabled: true
        description: "healthcheck failed 3 times in a row"
        send-on-resolved: true
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"

Here's an example of what the notifications look like:

Configuring Messagebird alerts

Example of sending SMS text message alert using Messagebird:

alerting:
  messagebird:
    access-key: "..."
    originator: "31619191918"
    recipients: "31619191919,31619191920"
services:
  - name: twinnation
    interval: 30s
    url: "https://twinnation.org/health"
    alerts:
      - type: messagebird
        enabled: true
        failure-threshold: 3
        send-on-resolved: true
        description: "healthcheck failed 3 times in a row"
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"

Configuring custom alerts

While they're called alerts, you can use this feature to call anything.

For instance, you could automate rollbacks by having an application that keeps tracks of new deployments, and by leveraging Gatus, you could have Gatus call that application endpoint when a service starts failing. Your application would then check if the service that started failing was recently deployed, and if it was, then automatically roll it back.

The values [ALERT_DESCRIPTION] and [SERVICE_NAME] are automatically substituted for the alert description and the service name respectively in the body (alerting.custom.body) as well as the url (alerting.custom.url).

If you have send-on-resolved set to true, you may want to use [ALERT_TRIGGERED_OR_RESOLVED] to differentiate the notifications. It will be replaced for either TRIGGERED or RESOLVED, based on the situation.

For all intents and purpose, we'll configure the custom alert with a Slack webhook, but you can call anything you want.

alerting:
  custom:
    url: "https://hooks.slack.com/services/**********/**********/**********"
    method: "POST"
    insecure: true
    body: |
      {
        "text": "[ALERT_TRIGGERED_OR_RESOLVED]: [SERVICE_NAME] - [ALERT_DESCRIPTION]"
      }
services:
  - name: twinnation
    url: "https://twinnation.org/health"
    interval: 30s
    alerts:
      - type: custom
        enabled: true
        failure-threshold: 10
        success-threshold: 3
        send-on-resolved: true
        description: "healthcheck failed 10 times in a row"
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"

Kubernetes (ALPHA)

WARNING: This feature is in ALPHA. This means that it is very likely to change in the near future, which means that while you can use this feature as you see fit, there may be breaking changes in future releases.

Parameter	Description	Default
`kubernetes`	Kubernetes configuration	`{}`
`kubernetes.auto-discover`	Whether to enable auto discovery	`false`
`kubernetes.cluster-mode`	Cluster mode to use for authenticating. Supported values: `in`, `out`	Required `""`
`kubernetes.service-template`	Service template. See `services[]` in Configuration	Required `nil`
`kubernetes.excluded-service-suffixes`	List of service suffixes to not monitor (e.g. `canary`)	`[]`
`kubernetes.namespaces`	List of configurations for the namespaces from which services will be discovered	`[]`
`kubernetes.namespaces[].name`	Namespace name	Required `""`
`kubernetes.namespaces[].hostname-suffix`	Suffix to append to the service name before calling `target-path`	Required `""`
`kubernetes.namespaces[].target-path`	Path that will be called on the discovered service for the health check	`""`
`kubernetes.namespaces[].excluded-services`	List of services to not monitor in the given namespace	`[]`

Auto Discovery

Auto discovery works by reading all Service resources from the configured namespaces and appending the hostname-suffix as well as the configured target-path to the service name and making an HTTP call.

All auto-discovered services will have the service configuration populated from the service-template.

You can exclude certain services from the dashboard by using kubernetes.excluded-service-suffixes or kubernetes.namespaces[].excluded-services.

kubernetes:
  auto-discover: true
  # out: Gatus is deployed outside of the K8s cluster.
  # in: Gatus is deployed in the K8s cluster
  cluster-mode: "out"                                              
  excluded-service-suffixes:
    - canary
  service-template:
    interval: 30s
    conditions:
      - "[STATUS] == 200"
  namespaces:
    - name: default
      # If cluster-mode is out, you should use an externally accessible hostname suffix (e.g.. .example.com)
      # This will result in gatus generating services with URLs like <service-name>.example.com
      # If cluster-mode is in, you can use either an externally accessible hostname suffix (e.g.. .example.com)
      # or an internally accessible hostname suffix (e.g. .default.svc.cluster.local)
      hostname-suffix: ".default.svc.cluster.local"
      target-path: "/health"
      # If some services cannot be or do not need to be monitored, you can exclude them by explicitly defining them
      # in the following list.
      excluded-services:
        - gatus
        - kubernetes

Note that hostname-suffix could also be something like .yourdomain.com, in which case the endpoint that would be monitored would be potato.example.com/health, assuming you have a service named potato and a matching ingress to map potato.example.com to the potato service.

Deploying

See example/kubernetes-with-auto-discovery

Docker

Other than using one of the examples provided in the examples folder, you can also try it out locally by creating a configuration file - we'll call it config.yaml for this example - and running the following command:

docker run -p 8080:8080 --mount type=bind,source="$(pwd)"/config.yaml,target=/config/config.yaml --name gatus twinproduction/gatus

If you're on Windows, replace "$(pwd)" by the absolute path to your current directory, e.g.:

docker run -p 8080:8080 --mount type=bind,source=C:/Users/Chris/Desktop/config.yaml,target=/config/config.yaml --name gatus twinproduction/gatus

Running the tests

go test ./... -mod vendor

Using in Production

See the example folder.

FAQ

Sending a GraphQL request

By setting services[].graphql to true, the body will automatically be wrapped by the standard GraphQL query parameter.

For instance, the following configuration:

services:
  - name: filter users by gender
    url: http://localhost:8080/playground
    method: POST
    graphql: true
    body: |
      {
        user(gender: "female") {
          id
          name
          gender
          avatar
        }
      }
    headers:
      Content-Type: application/json # XXX: as of v1.9.2, this header is automatically added when graphql is set to true
    conditions:
      - "[STATUS] == 200"
      - "[BODY].data.user[0].gender == female"

will send a POST request to http://localhost:8080/playground with the following body:

{"query":"      {\n        user(gender: \"female\") {\n          id\n          name\n          gender\n          avatar\n        }\n      }"}

Recommended interval

NOTE: This does not really apply if disable-monitoring-lock is set to true, as the monitoring lock is what tells Gatus to only evaluate one service at a time.

To ensure that Gatus provides reliable and accurate results (i.e. response time), Gatus only evaluates one service at a time In other words, even if you have multiple services with the exact same interval, they will not execute at the same time.

You can test this yourself by running Gatus with several services configured with a very short, unrealistic interval, such as 1ms. You'll notice that the response time does not fluctuate - that is because while services are evaluated on different goroutines, there's a global lock that prevents multiple services from running at the same time.

Unfortunately, there is a drawback. If you have a lot of services, including some that are very slow or prone to time out (the default time out is 10s for HTTP and 5s for TCP), then it means that for the entire duration of the request, no other services can be evaluated.

This does mean that Gatus will be unable to evaluate the health of other services. The interval does not include the duration of the request itself, which means that if a service has an interval of 30s and the request takes 2s to complete, the timestamp between two evaluations will be 32s, not 30s.

While this does not prevent Gatus' from performing health checks on all other services, it may cause Gatus to be unable to respect the configured interval, for instance:

Service A has an interval of 5s, and times out after 10s to complete
Service B has an interval of 5s, and takes 1ms to complete
Service B will be unable to run every 5s, because service A's health evaluation takes longer than its interval

To sum it up, while Gatus can really handle any interval you throw at it, you're better off having slow requests with higher interval.

As a rule of the thumb, I personally set interval for more complex health checks to 5m (5 minutes) and simple health checks used for alerting (PagerDuty/Twilio) to 30s.

Default timeouts

Protocol	Timeout
HTTP	10s
TCP	5s

Monitoring a TCP service

By prefixing services[].url with tcp:\\, you can monitor TCP services at a very basic level:

services:
  - name: redis
    url: "tcp://127.0.0.1:6379"
    interval: 30s
    conditions:
      - "[CONNECTED] == true"

Placeholders [STATUS] and [BODY] as well as the fields services[].body, services[].insecure, services[].headers, services[].method and services[].graphql are not supported for TCP services.

NOTE: [CONNECTED] == true does not guarantee that the service itself is healthy - it only guarantees that there's something at the given address listening to the given port, and that a connection to that address was successfully established.

Monitoring a service using ICMP

By prefixing services[].url with icmp:\\, you can monitor services at a very basic level using ICMP, or more commonly known as "ping" or "echo":

services:
  - name: ICMP
    url: "icmp://example.com"
    conditions:
      - "[CONNECTED] == true"

Only the placeholders [CONNECTED], [IP] and [RESPONSE_TIME] are supported for services of type ICMP. You can specify a domain prefixed by icmp://, or an IP address prefixed by icmp://.

Monitoring a service using DNS queries

Defining a dns configuration in a service will automatically mark that service as a service of type DNS:

services:
  - name: example dns query
    url: "8.8.8.8" # Address of the DNS server to use
    interval: 30s
    dns:
      query-name: "example.com"
      query-type: "A"
    conditions:
      - "[BODY] == 93.184.216.34"
      - "[DNS_RCODE] == NOERROR"

There are two placeholders that can be used in the conditions for services of type DNS:

The placeholder [BODY] resolves to the output of the query. For instance, a query of type A would return an IPv4.
The placeholder [DNS_RCODE] resolves to the name associated to the response code returned by the query, such as NOERROR, FORMERR, SERVFAIL, NXDOMAIN, etc.

Basic authentication

You can require Basic authentication by leveraging the security.basic configuration:

security:
  basic:
    username: "john.doe"
    password-sha512: "6b97ed68d14eb3f1aa959ce5d49c7dc612e1eb1dafd73b1e705847483fd6a6c809f2ceb4e8df6ff9984c6298ff0285cace6614bf8daa9f0070101b6c89899e22"

The example above will require that you authenticate with the username john.doe as well as the password hunter2.

disable-monitoring-lock

Setting disable-monitoring-lock to true means that multiple services could be monitored at the same time.

While this behavior wouldn't generally be harmful, conditions using the [RESPONSE_TIME] placeholder could be impacted by the evaluation of multiple services at the same time, therefore, the default value for this parameter is false.

There are three main reasons why you might want to disable the monitoring lock:

You're using Gatus for load testing (each services are periodically evaluated on a different goroutine, so technically, if you create 100 services with a 1 seconds interval, Gatus will send 100 requests per second)
You have a lot of services to monitor
You want to test multiple services at very short interval (< 5s)

Service groups

Service groups are used for grouping multiple services together on the dashboard.

services:
  - name: frontend
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

  - name: backend
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

  - name: monitoring
    group: internal
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

  - name: nas
    group: internal
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

  - name: random service that isn't part of a group
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

The configuration above will result in a dashboard that looks like this:

Exposing Gatus on a custom port

By default, Gatus is exposed on port 8080, but you may specify a different port by setting the web.port parameter:

web:
  port: 8081

If you're using a PaaS like Heroku that doesn't let you set a custom port and exposes it through an environment variable instead, you can use that environment variable directly in the configuration file:

web:
  port: ${PORT}

Uptime badges

Gatus can automatically generate a SVG badge for one of your monitored services. This allows you to put badges in your individual services' README or even create your own status page, if you desire.

The endpoint to generate a badge is the following:

/api/v1/badges/uptime/{duration}/{identifier}.svg

Where:

{duration} is 7d, 24h or 1h
{identifier} has the pattern <GROUP_NAME>_<SERVICE_NAME>.svg in which both variables have , /, _, , and . replaced by -.

For instance, if you want the uptime during the last 24 hours from the service frontend in the group core, the URL would look like this:

http://example.com/api/v1/badges/uptime/7d/core_frontend.svg

If you want to display a service that is not part of a group, you must leave the group value empty:

http://example.com/api/v1/badges/uptime/7d/_frontend.svg

Example:

![Uptime 24h](https://status.twinnation.org/api/v1/badges/uptime/24h/core_twinnation-external.svg)

If you'd like to see a visual example of each badges available, you can simply navigate to the service's detail page.

Owner

Chris C.

I love programming

https://github.com/TwinProduction/gatus https://status.twinnation.org/

Comments

Add html content check condition

As mentioned in my previous issue #61 i'm currently using healthchecks, and the last feature i would need is some generic content regex / string presence check. now, i might be missing something everybody already knows, since you have this BODY placeholder, and maybe that's just it.

anyway, it seems that the process basically would mean gatus curls the site content and runs a regex or literal string against it, and when it finds it, test passes.

what do you think?

in terms of performance, i think that this is actually the most heavy that healthchecks is doing, but already now i'm running all the tests in parallel, and gatus outperforms healthchecks by magnitudes. so nice.
GoogleChat alerts are blank under certain conditions for ICMP endpoints

Describe the bug

When using ICMP and [CONNECTED] == true, whenever an endpoint fails, the Google Chat/Spaces App only sends a blank message with no data.

What do you see?

Nothing is displayed, it sends an empty message.

What do you expect to see?

Full alert, like with HTTPS, which works just fine. This is ICMP only.

List the steps that must be taken to reproduce this issue

Setup googlechat alert, have an ICMP endpoint fail.

Version

latest branch

Additional information

No response
Redirection checks are not working

First of all, thanks for such a simple tool and for making it open source :) I tried to see if the following issue was posted already, but I couldn't find anything.

All my 301 and 302 checks are failing, and it seems to be because when making the HTTP requests, it's following redirects by default.

Here's an example. https://inbox.google.com does a 301 to https://mail.google.com/, but the status code in the image below is 200.

Is this by design?
Scalability limits?

I'm monitoring around 50-55 services with gatus, most are HTTP and 29 of them are using the pat keyword (with wildcards, so about as expensive a query as it can get). All using the default poll interval of 60s.

I am starting to see some responses of context deadline exceeded (Client.Timeout exceeded while awaiting headers) in the body check. I have manually checked the services in question and they're healthy. Restarting gatus, or just waiting a few minutes seems to resolve this. This does not occur continuously, but have seen it twice in the space of a few hours.

I can only assume that this is due to a concurrency issue, as it's more than possible that the combination of service times takes longer than 60s to respond. I do not know enough of the gatus architecture to know if this is a problem or not.

I am running v2.3.0 with #100 changed locally (as I've not updated since I tested it). I will repeat the test with 2.4.0 and report the results here.

Support grouping services

Supporting service groups could allow a cuter front end experience.

i.e.

services:
  - name: k8s-cluster-watch-dog
    url: http://k8s-cluster-watch-dog-v1.tools-${ENVIRONMENT}:8080/health
    group: core         <-------
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
  - name: prometheus
    url: http://prometheus-operator-prometheus.kube-system:9090/-/healthy
    group: core         <-------
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      - "[BODY] == Prometheus is Healthy."

would generate a dashboard that puts both k8s-cluster-watch-dog and prometheus under the "core" folder.

Could also support tags, and allow filtering by tags instead

i.e.

services:
  - name: k8s-cluster-watch-dog
    url: http://k8s-cluster-watch-dog-v1.tools-${ENVIRONMENT}:8080/health
    tags:              <-------
      - core
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
  - name: prometheus
    url: http://prometheus-operator-prometheus.kube-system:9090/-/healthy
    tags:              <-------
      - core
      - metrics
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      - "[BODY] == Prometheus is Healthy."

Forgive the terrible drafts, just thought of this on the fly.

ICMP does not work on Mac OS
MAC OS Big Sur version 11.0.1

Everything works perfectly ok except icmp in MAC OS.

Config:

name: node-1 url: "icmp://104.237.x.x" group: Uptime (Ping) conditions:

"[CONNECTED] == true"

"[RESPONSE_TIME] > 0"

the x.x is in ip is set intentionally.

Is this wrong config?

Support wecom alerting provider

Hello, I'm a Chinese and my team use Wechat Company (Wecom) to watch status of servers.

I'm not a developer, I try to write wecom alerting code and it work's fine on my machine. But because of my Go language level, there may be many exceptions that have not been handled. If possible, I hope it can be merge into the official code. Thanks.

Push to Wecom, we need 3 necessary parameters, Aid, Cid and Secret.

package wecom

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"

	"github.com/TwinProduction/gatus/alerting/alert"
	"github.com/TwinProduction/gatus/alerting/provider/custom"
	"github.com/TwinProduction/gatus/core"
)

// AlertProvider is the configuration necessary for sending an alert using Wecom
type AlertProvider struct {
	Aid    string `yaml:"aid"`
	Cid    string `yaml:"cid"`
	Secret string `yaml:"secret"`
	ToUser string `yaml:"touser"` // Not necessary
	ToParty string `yaml:"toparty"` // Not necessary

	// DefaultAlert is the default alert configuration to use for services with an alert of the appropriate type
	DefaultAlert *alert.Alert `yaml:"default-alert"`
}

Then, we need use Cid and Secret to get accessToken.

	var wetoken string
	wetoken = fmt.Sprintf("%s%s%s%s", "https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=", provider.Cid, "&corpsecret=", provider.Secret)
	client := &http.Client{}
	req, err := http.NewRequest("GET", wetoken, nil)
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println("Failure : ", err)
	}
	respBody, _ := ioutil.ReadAll(resp.Body)
	r := make(map[string]interface{})
	json.Unmarshal([]byte(respBody), &r)
	accessToken := r["access_token"]

Last, use accessToken to push.

	return &custom.AlertProvider{
		URL:    fmt.Sprintf("https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=%s", accessToken),
		Method: http.MethodPost,
		Body: fmt.Sprintf( TODO:BODY ),
		Headers: map[string]string{"Content-Type": "application/json"},
	}

If ToUser and ToParty are empty, the BODY will be:

{
	"agentid": %s,
	"msgtype": "text",
	"text": {
		"content": "%s"
	},
	"duplicate_check_interval": 180
}

If one of the two is empty, the BODY will be:

{
	"toparty": %s,
	"agentid": %s,
	"msgtype": "text",
	"text": {
		"content": "%s"
	},
	"duplicate_check_interval": 180
}

{
	"touser": %s,
	"agentid": %s,
	"msgtype": "text",
	"text": {
		"content": "%s"
	},
	"duplicate_check_interval": 180
}

If both parameters are not empty, the BODY will be:

{
	"touser": %s,
	"toparty": %s,
	"agentid": %s,
	"msgtype": "text",
	"text": {
		"content": "%s"
	},
	"duplicate_check_interval": 180
}

Implement Postgres as a storage solution

I stumbled on this neat status page solution the other day and am trying it out, cool stuff!

I see Store is an interface, and that there is one implementation (in-memory cache with a file used to persist over restarts). However I think it's missing some way to persist the observed results which is more usable in a stateless container, where you don't want to write to a file system.

Is there perhaps already some extension out there that provides a Store backed by e.g. Redis or Postgres? I would not mind contributing, but if the effort had already been made then that would be unnecessary.

The point would not be to enable some high availability configuration (I think that would add quite a bit more synchronization work), but simply being able to restart or reconfigure gatus without losing the previous results would be nice.

Also, the auto save interval could be made configurable.
feat(metrics): Add more metrics (e.g. duration)

Describe the feature request

In this project https://github.com/prometheus/blackbox_exporter, many metrics like probe_success, probe_duration etc... provided. And in dns probe https://github.com/prometheus/blackbox_exporter/blob/master/prober/dns.go#L129 also has metrics for dns.

Why do you personally want this feature to be implemented?

Maybe we can add these metrics in gatus, and user can scrpe to promethus and make dashboard on Grafana.

How long have you been using this project?

3 month

Additional information

If the team agrees to add this feature, I can submit PR to contribute gatus
Change license to Apache 2
Call me indecisive if you want, since I've done this once before (see 70c9c4b87c8e595f4374b895853312cefc2152f2), but after thing about the pros and cons, I decided that Apache 2 offers better protection for Gatus.

Long story short:

v0.0.1 to v1.3.0: Apache 2

v1.3.1 to v3.3.5: MIT

v3.3.6 and up: Apache 2

This PR is here for traceability.
Bind mounted config.yaml does not automatically reload
Step to reproduce:

$ cat config/config.yaml services: - name: example url: "https://example.org/" interval: 30s conditions: - "[STATUS] == 200"

$ docker run \ --rm \ -p 8080:8080 \ -v "$(pwd)"/config/config.yaml:/config/config.yaml \ twinproduction/gatus

Change the name field in the config/config.yaml.

Observation:

In the HasLoadedConfigurationFileBeenModified function, the result of the comparison of config.lastFileModTime.Unix() != fileInfo.ModTime().Unix() is always false.
feat: API for adding or removing configuration

Describe the feature request

I'd like Gatus to have an API that allows managing its configuration so I can create a Kubernetes controller (or maybe extend https://github.com/stakater/IngressMonitorController) to watch Ingress resources and automatically create checks in Gatus for them.

An alternative could be to leverage something like https://github.com/TwiN/gatus/issues/326 and have a sidecar like the https://github.com/kiwigrid/k8s-sidecar to pass the configuration to Gatus but that would have more moving parts.

Why do you personally want this feature to be implemented?

To be able to programatically setup checks for services running in a Kuberntes cluster.

How long have you been using this project?

No response

Additional information

No response
Allow configuration to be distributed
Summary

This PR allows configuration to split across multiple files using the common config.d/-path style. The approach reads all *.yml and *.yaml files and merges them in memory into a single file before parsing. This seemed easier und required fewer changes than parsing each file individually and merging the objects afterwards.

Fixes #326

As I did not work with golang before, feel free to remark on obvious errors and bad patterns.

I envisaged to write tests, but did not figure out yet, how invasive tests are intended to be structured for this project. I hope the chosen way is in you interest.

Checklist

[x] Tested and/or added tests to validate that the changes work as intended, if applicable.

[x] Added the documentation in README.md, if applicable.
bug: Investigate flaky `TestStore_InsertCleansUpOldUptimeEntriesProperly` test
Describe the bug

Sometimes, the TestStore_InsertCleansUpOldUptimeEntriesProperly test will fail

What do you see?

https://github.com/TwiN/gatus/actions/runs/3723336201/attempts/1

--- FAIL: TestStore_InsertCleansUpOldUptimeEntriesProperly (0.04s) sql_test.go:109: oldest endpoint uptime entry should've been ~5 hours old, was 4h59m40.064240509s sql_test.go:119: oldest endpoint uptime entry should've been ~5 hours old, was 4h59m40.068831372s sql_test.go:129: oldest endpoint uptime entry should've been ~8 hours old, was 7h59m40.073425836s sql_test.go:139: oldest endpoint uptime entry should've been ~239h0m0s hours old, was 238h59m40.0780609s sql_test.go:150: oldest endpoint uptime entry should've been ~8 hours old, was 7h59m40.08307097s

What do you expect to see?

Test should pass

List the steps that must be taken to reproduce this issue

Execute TestStore_InsertCleansUpOldUptimeEntriesProperly between hh:59:30 and hh:59:59

Note how the test will fail

Version

latest (master)

Additional information

I suspect this has something to do with https://github.com/TwiN/gatus/blob/f6a621da285dd43b3207ec21e4101b4b11eff6c5/storage/store/sql/sql_test.go#L100
feat: repeating notifications?

Describe the feature request

Ability to send repeated notifications every x min/hr if the endpoint is still offline/dead.

Why do you personally want this feature to be implemented?

We now had over 300 nodes monitoring via Gatus and we received the notifications via Slack and Telegram. One scenario is when some nodes are offline during the weekend. We always forget to fix them in the next workday.

I've tried some other services and seems like it's a common feature to alert repeatedly if something still goes wrong.

How long have you been using this project?

No response

Additional information

No response
Unclear error message `invalid character '<' looking for beginning of value`
Describe the bug

A JSON API endpoint I'm monitoring briefly fails from time to time with the error message

invalid character '<' looking for beginning of value

The failing condition is len([BODY].25447.bill) >= 1.

I don't really understand what the error message is supposed to mean. No JSON payload present in the [BODY]?

What do you see?

An opaque error message.

What do you expect to see?

A clear/understandable error message.

List the steps that must be taken to reproduce this issue

Set up an endpoint with

- name: "TEST" enabled: true url: "https://api.votelog.ch/api/v1/bill/380b49bf-8044-4f65-a2e5-ac3a00e3e083/votes?lang=de" interval: 60s conditions: - "[STATUS] == 200" - "len([BODY].25447.bill) >= 1"

Version

4.3.2

Additional information

No response
Allow to set separate logo for dark mode

Describe the feature request

When using a black-and-white or dark logo with transparent background[*], it becomes barely visible when dark mode is toggled on. The same is true for light mode with a white/light logo with transparent background. Thus it would be nice if we could configure two seperate logo files/URLs, one for the (default) light mode and one for the dark mode.

Config could become ui.logo.light and ui.logo.dark, maybe with a fallback to the current behaviour when ui.logo is directly set to a string instead of the two subkeys.

Why do you personally want this feature to be implemented?

It would improve Gatus' web frontend aesthetically. 💅

How long have you been using this project?

3 days (awesome project! ❤️)

Additional information

[*] Example Gatus instance with such a logo is found here: https://status.votelog.ch/

⛑ Gatus - Automated service health dashboard

Table of Contents

Features

Usage

Configuration

Conditions

Placeholders

Functions

Alerting

Configuring Slack alerts

Configuring PagerDuty alerts

Configuring Twilio alerts

Configuring Mattermost alerts

Configuring Messagebird alerts

Configuring custom alerts

Kubernetes (ALPHA)

Auto Discovery

Deploying

Docker

Running the tests

Using in Production

FAQ

Sending a GraphQL request

Recommended interval

Default timeouts

Monitoring a TCP service

Monitoring a service using ICMP

Monitoring a service using DNS queries

Basic authentication

disable-monitoring-lock

Service groups

Exposing Gatus on a custom port

Uptime badges

Owner

Chris C.

Comments

Add html content check condition

GoogleChat alerts are blank under certain conditions for ICMP endpoints

Describe the bug

What do you see?

What do you expect to see?

List the steps that must be taken to reproduce this issue

Version

Additional information

Redirection checks are not working

Scalability limits?

Support grouping services

ICMP does not work on Mac OS

Support wecom alerting provider

Implement Postgres as a storage solution

feat(metrics): Add more metrics (e.g. duration)

Describe the feature request

Why do you personally want this feature to be implemented?

How long have you been using this project?

Additional information

Change license to Apache 2

Bind mounted config.yaml does not automatically reload

feat: API for adding or removing configuration

Describe the feature request

Why do you personally want this feature to be implemented?

How long have you been using this project?

Additional information

Allow configuration to be distributed

Summary

Checklist

bug: Investigate flaky `TestStore_InsertCleansUpOldUptimeEntriesProperly` test

Describe the bug

What do you see?

What do you expect to see?

List the steps that must be taken to reproduce this issue

Version

Additional information

feat: repeating notifications?

Describe the feature request

Why do you personally want this feature to be implemented?

How long have you been using this project?

Additional information

Unclear error message `invalid character '<' looking for beginning of value`

Describe the bug

What do you see?