Add:

• buffered channel writes
• mechanism to recover from when all channel buffers are full
• internal metrics to give additional insight into the stability of the service
• documentation on internal metrics
• enable "Nagle's algorithm" for output to consumers to improve network performance

This change will improve the service's stability, speed, and observability. However, the changes may increase latency and delays for the metrics consumers due to the added buffering.

Currently, the graphite metrics path for internal Hadrianus metrics are available on a hard-coded metrics path. However, to suit how different organizations or individuals organize their metrics, it should be possible to customize where you can find the metrics. The most flexible way to accomplish this could be to specify this on the command line (or configuration) using Go templates.

It becomes very choppy in many of our graphs where the value rarely changes:

We can solve the problem like this:

• Remember how long it took before Hadrianus removed the value from the "stale" state.
• Use the stale time interval to decide how much additional time should pass until the metric is placed in a "stale" state next time.

It may also be a good idea to set a maximum "additional stale time" limit. When the limit is 0, the feature is disabled.

Since one of the current uses of Hadrianus is to send metrics to Grafana Cloud, it might simplify the setup to add Grafana Cloud support to Hadrianus. That way, we'd be able to spin up Hadrianus as a scratch container in EKS with minimal fuss, without using carbon-relay-ng.

The integration of the code seems super simple: https://github.com/grafana/cloud-graphite-scripts/blob/master/send/main.go

The above example uses "plain json" (content-type "application/json"), which is pretty inefficient. Instead, it should use "binary protocol, snappy compressed" (content-type "rt-metric-binary-snappy"). Some work needs to be spent on understanding how this is done.

We still need to think about how the UI/config should look for it (config maps? env variables?).

One way to handle this is to define named output aliases in a configuration file. It overrides the standard resolve hostname/port mechanism when such a name is encountered. Each output alias needs to have:

• name
• type (for example "grafanaNet" or "plaintext")
• optional configuration details (for example, "address URL", "API key", "schemas file", aggregation file" for "grafanaNet")

Internal Hadrianus metrics should also (optionally?) be exposed as a Prometheus endpoint. This could have the additional benefit of being usable as a service health check in Kubernetes or similar.

server.hadrianus.*.clientConnectionsActive will sometimes drift out of sync and give strange results.

Possibly, this could be fixed by replacing the existing naïve counters with "atomic counters" (https://gobyexample.com/atomic-counters).

It can be problematic for a graphite setup if metrics producers create new metrics paths at a high rate.

There should be a way to limit the creation of new metrics paths for a metrics producer to stop the downstream systems from being overwhelmed.

Suggestions:

• The production of new metrics should be possible to throttle by producer IP address.
• Once throttled, a producer should only be allowed to create a limited number of new metrics paths per measurement period.

A mock example of keeping track of this could look like this. The first level is a 128-bit IPv6 address. The second level contains the 32 bit hash of a graphite metrics path, mapped to a 64 bit UNIX timestamp. The idea is that every N seconds (whatever your measurement time interval is), you count how many paths have a timestamp that is greater than or equal to the current timestamp minus N for every IP address. This approach will give you the number of unique metric paths produced for the time interval N for every IP address:

{
    "::ffff:90.16.154.34": {
        "c65a134b": 1642598151,
        "c296b298": 1642598151
    },
    "dc8b:334c:b60c:d107:47d3:ca23:2860:df1a": {
        "dd7ee3a6": 1642598153,
        "a3cf9f1b": 1642598153,
        "d4c8af8d": 1642598153
    }
}

The internal Hadrianus metric server.hadrianus.*.clientConnectionsActive isn't beneficial when you have a small number of connections, since it's just sampling the number of connections currently.

It should (perhaps) instead of this, show an average of the number of concurrent active network connections during the sampling interval.