I noticed a few grafana display bugs in my graphs which at first I blamed on Thanos but forgot I had a trickster in front of it.

Expected graph

Thanos + Trickster graph

The data are the same but only come from different origin/path.

This only happens when I set the end of range to now in grafana. If fix the end of range to a fixed time I can't reproduce.

Taking a look at the data coming from Prometheus and Thanos + Trickster I got the following diff

As you can see the last timestamp is not aligned with the step in the Thanos + Trickster case and it confuses grafana's output.

Bypassing Trickster fixes the problem.

If I send a simple scalar query such as /api/v1/query?query=5 trickster now returns errors. I bisected this and it seems that the error was introduced in ec4eff34d5532b1907723eeaabe620f02dd25b32 . The basic problem here is that trickster assumes the response type from prometheus as a vector, when in reality it can be (1) scalar (2) vector or (3) string. The way this worked before was that the unmarshal error was ignored and the response handed back (I have a patch that puts it back to that behavior.

From what I see that looks like caching won't work on those scalar values -- as it wasn't able to unmarshal. Alternatively we could change the model to use model.Value and then add some type-switching. But since that is a potentially large change I figured I'd get others' input first.

After configuring Trcikster to use AWS ElastiCache/Redis 5.0.4, deployed as Master + 2 Replicas and without cluster mode enabled.

Using Trickster 1.0-beta8 image

Deployed on K8s with 3 replicas and exposing Trickester through an Ingress to allow Grafana to use it as Datasource.

Every time Grafana runs a query Trickster misses the cache, even copying the request from Grafana and running it with curl multiple times it keeps failing the cache

    [caches]

        [caches.default]
        # cache_type defines what kind of cache Trickster uses
        # options are 'bbolt', 'filesystem', 'memory', 'redis' and 'redis_cluster'
        # The default is 'memory'.
        type = 'redis'

        # compression determines whether the cache should be compressed. default is true
        # changing the compression setting will leave orphans in your cache for the duration of timeseries_ttl_secs
        compression = true

        # timeseries_ttl_secs defines the relative expiration of cached timeseries. default is 6 hours (21600 seconds)
        timeseries_ttl_secs = 21600

        # fastforward_ttl_secs defines the relative expiration of cached fast forward data. default is 15s
        fastforward_ttl_secs = 15

        # object_ttl_secs defines the relative expiration of generically cached (non-timeseries) objects. default is 30s
        object_ttl_secs = 30

            ### Configuration options for the Cache Index
            # The Cache Index handles key management and retention for bbolt, filesystem and memory
            # Redis handles those functions natively and does not use the Trickster's Cache Index
            [caches.default.index]

            # reap_interval_secs defines how long the Cache Index reaper sleeps between reap cycles. Default is 3 (3s)
            reap_interval_secs = 3

            # flush_interval_secs sets how often the Cache Index saves its metadata to the cache from application memory. Default is 5 (5s)
            flush_interval_secs = 5

            # max_size_bytes indicates how large the cache can grow in bytes before the Index evicts least-recently-accessed items. default is 512MB
            max_size_bytes = 536870912

            # max_size_backoff_bytes indicates how far below max_size_bytes the cache size must be to complete a byte-size-based eviction exercise. default is 16MB
            max_size_backoff_bytes = 16777216

            # max_size_objects indicates how large the cache can grow in objects before the Index evicts least-recently-accessed items. default is 0 (infinite)
            max_size_objects = 0

            # max_size_backoff_objects indicates how far under max_size_objects the cache size must be to complete object-size-based eviction exercise. default is 100
            max_size_backoff_objects = 100



            ### Configuration options when using a Redis Cache
            [caches.default.redis]
            # protocol defines the protocol for connecting to redis ('unix' or 'tcp') 'tcp' is default
            protocol = 'tcp'
            # endpoint defines the fqdn+port or path to a unix socket file for connecting to redis
            # default is 'redis:6379'
            endpoint = 'redis-common.external-service.svc.cluster.local:6379'
            # password provides the redis password
            # default is empty
            password = ''

Setting value_retention_factor = 1536 to conf file has no impact. Only responses that have less than default 1024 sample are fully cached. Config file is correctly used because I can switch to filesystem caching.

This is issue when Grafana Prometheus datasource use Resolution 1/1. It result small possible range_query "step", sample per pixel. For example querying 24h data the step is set to 60s --> 1441 sample returned and only 1024 are cached resulting always kmiss and phit.

Workaround: When Grafana resolution is set to 1/2 --> step size increase to 120s --> 721 sample returned and Trickster has 100% cache hit rate.

Hi,

I installed v0.1.3 in order to see if it fixes https://github.com/Comcast/trickster/issues/92 but I encountered this.

panic: runtime error: index out of range
goroutine 493 [running]:
main.(*PrometheusMatrixEnvelope).cropToRange(0xc0001c06f8, 0x5c06535c, 0x0)
	/go/src/github.com/Comcast/trickster/handlers.go:1014 +0x4ee
main.(*TricksterHandler).originRangeProxyHandler(0xc000062140, 0xc000312640, 0x41, 0xc00014c8a0)
	/go/src/github.com/Comcast/trickster/handlers.go:840 +0x615
created by main.(*TricksterHandler).queueRangeProxyRequest
	/go/src/github.com/Comcast/trickster/handlers.go:660 +0x276

I'm super excited about this project! Thanks for sharing it with the community!

I had a question about this part of the docs:

In a Multi-Origin placement, you have one dashboard endpoint, one Trickster endpoint, and multiple Prometheus endpoints. Trickster is aware of each Prometheus endpoint and treats them as unique databases to which it proxies and caches data independently of each other.

Could this work for load balancing multiple Prometheus servers in a HA setup? We currently have a pair of Prometheus servers in each region, redundantly scraping the same targets. Currently our Grafana is just pinned to one Prometheus server in each region, meaning that if that one goes down our dashboards go down until we manually change the datasource to point to the other one (and by that point we would have just restored the first server anyway). It's kind of a bummer because it means that while HA works great for alerting itself, it doesn't work for dashboards.

Would be awesome if there was a way to achieve this with Trickster!

I faced an issue yesterday with Chart 1.1.2, it seems the 1.0-beta chaged to be copy of 1.0-beta10 which caused trickster to fail to start after updating my kubernetes cluster nodes and pulling the docker image. then I tried to update to latest chart.

for 1.3.0 also some values in values.yaml is duplicated like service section which cause an issue with config file having an empty listen_port

also PVC contains deprecated values for example https://github.com/Comcast/trickster/blob/6ac009f29aeeea9476da9db6311d0aa7cf39033c/deploy/helm/trickster/templates/pvc.yaml#L1 there is no section called config in current values.yaml

workaround: used tag 1.0.9-beta9

We're using the redis cache backend in our instance of trickster and we're seeing suprisingly high memory usage.

We are running trickster in kubernetes, with memory limited to 4GB. Trickster only needs to be up for half an hour before it's killed for using more than 4GB memory (OOM).

Has anyone seen any similar behavior? Any idea what could be going on? We've tested increasing the memory limits, they were originally set to 2GB, but the problem persists.

It seems like when we load test trickster with a prometheus backend, that 1.1 has a reasonably large performance regression as compared to 1.0

both 1.1.2 and 1.1.3 seems to max out at around 100 requests per second when we load test it. 1.0 doesn't seem to get throttled in the same way and can go to several hundred (haven't tried higher yet). We also see much higher cpu and memory usage for 1.1.

We're trying about 15 different queries set to query over the last hour.

This method is not very large, but does a few things which IMO are questionable.

1. reversing start/end if the user flipped start/end, it is not the responsibility of trickster to flip it. This is additionally confusing as things will work, but if trickster is removed from the call path all queries this fixes will break. IMO it is not the place of the cache to "correct" user queries.

2. time.Now() checks I think its fine to cut off queries from going into the future, but this doesn't handle the case where both start/end are in the future. IMO in that case this should return an error. In the remaining cases I think its fine to leave it truncating end as the query results will remain unaffected, although I'd rather it didn't (the time.Now() constraint is a cache issue, not a query issue).

3. default step param If the user didn't define one, this is not the place of a caching proxy to correct it. Here we should just return an error and make the user correct their query.

Hi - I've set trickster to cache to redis, I'm using beta 8 (but have tried multiple versions). Trickster seems to connect fine and even tries to store data:

-- time=2019-06-19T12:56:57.101537228Z app=trickster caller=proxy/engines/cache.go:71 level=debug event="compressing cached data" cacheKey=thanos-query:9090.0a9332e4c9046613a62ba8a6e4a2e78a.sz time=2019-06-19T12:56:57.101899271Z app=trickster caller=cache/redis/redis.go:82 level=debug event="redis cache store" key=thanos-query:9090.0a9332e4c9046613a62ba8a6e4a2e78a.sz

However i get no keys in redis and the dashboard(s) load no quicker - I have tried Trickster with the in-memory option and this works as expected.

I am also able to write to redis both using the CLI and an external test application just to rule redis out.

I've also tried standing up multiple Redis types (e.g standard, cluster and sentinel)

Thanks!

https://github.com/trickstercache/trickster/blob/2eeb4ba048ed1676105cf954c849a39278ce38cc/pkg/proxy/listener/listener.go#L185-L189

This calls f() to exitFatal() but the log message does not show on the terminal as the loggers are not flushed. I wasted an hour trying to understand why the program won't start. Turns out I had a port already bind issue from a previous run but the log message won't print and since exitFatal() does not show stack trace there is no way to know what happened.

https://github.com/trickstercache/trickster/blob/main/pkg/proxy/tls/options/options.go#L79-L109

This validation check requires TLS is used for both frontend and backend. It should be possible to only use tls for backend and not use tls for frontend.

Trickster Version

Trickster version: 1.1.5, buildInfo: 2022-11-10T14:56:50+0000 , goVersion: go1.17.12, copyright: © 2018 Comcast Corporation

Problem

Every request made to an influxdb origin is returning X-Trickster-Result: engine=HTTPProxy; status=proxy-only.

Below is my configuration (left) and 2 contiguous attempts at querying trickster (right):

Is there something I am doing incorrectly in my configuration?

On that note - I've been trying to get Trickster to work for ClickHouse, Prometheus, and InfluxDB, and have only gotten it to work with Prometheus. Are there plans to maintain this project more actively? I've been told that the ClickHouse plugin does not work, and in general it would be great if the logging was improved to explain why certain states like proxy-only occur.

My company is at a point where we're considering new tools to help load balancing and caching toward ClickHouse, Prometheus, and Influx. Trickster appears to perfectly satisfy of our requirements based on what I've read, and it would be incredibly useful to for our use case. Thank you for your work so far!

Adds the capability from IronDB to cache SELECT requests in the short term to InfluxDB. Support for InfluxDB 2.0 and Flux incoming as part of these changes; draft to keep changes public & up to date, since this is one of the roadmap items.

Quick notes on changes:

• Purge by key uses Gorilla mux with path matching, ex /trickster/purge/key/{backend}/{key}. Purge by path uses http.ServeMux with query parameters, where the path is URL encoded, ex `/trickster/purge/path?backend={backend}&path={path}. This is to help manage URL encoded items, and to avoid altering too much of the existing admin router code.
• Purging by path uses a recreation of cache key derivation for requests, calculating an MD5 sum of the passed path.