Time Series in Go and PostgreSQL

Tgres is a program comprised of several packages which together can be used to receive, store and present time-series data using a relational database as persistent storage (currently only PostgreSQL).

You can currently use the standalone Tgres daemon as Graphite-like API and Statsd replacement all-in-one, or you can use the Tgres packages to incorporate time series collection and reporting functionality into your application.

See GoDoc for package details.

Whether you use standalone Tgres or as a package, the time series data will appear in your database in a compact and efficient format (by default as a view called tv), while at the same time simple to process using any other tool, language, or framework because it is just a table (or a view, rather). For a more detailed description of how Tgres stores data see this article

Current Status

Feb 7 2018: This project is not actively maintained. You may find quite a bit of time-series wisdom here, but there are probably still a lot of bugs.

Jul 5 2017: See this status update

Jun 15 2017: Many big changes since March, most notably data point versioning and instoduction of ds_state and rra_state tables which contain frequently changed attributes as arrays, similar to the way data points are stored eliminating the need to update ds and rra tables, these are now essentially immutable. Ability to delete series with NOTIFY to Tgres to purge it from the cache.

Mar 22 2017: Version 0.10.0b was tagged. This is our first beta (which is more stable than alpha). Please try it out, and take a minute to open an issue or even a PR if you see/fix any problems. Your feedback is most appreciated!

Feb 2017 Note: A major change in the database structure has been made, Tgres now uses the "write optimized" / "vertical" storage. This change affected most of the internal code, and as far overall status, it set us back a bit, all tests are currently broken, but on the bright side, write performance is amazing now.

Phase 1 or proof-of-concept for the project is the ability to (mostly) act as a drop-in replacement for Graphite (except for chart generation) and Statsd. Currently Tgres supports nearly all of Graphite functions.

As of Aug 2016 Tgres is feature-complete for phase 1, which means that the development will focus on tests, documentation and stability for a while.

Tgres is not ready for production use, but is definitely stable enough for tinkering for those interested.

Getting Started

You need a newer Go (1.7+) and PostgreSQL 9.5 or later. To get the daemon compiled all you need is:

$ go get github.com/tgres/tgres

Now you should have a tgres binary in $GOPATH/bin.

There is also a Makefile which lets you build Tgres with make which will use a slightly more elaborate command and the resulting tgres binary will be able to report its build time and git revision, but otherwise it's the same.

Look in $GOPATH/src/github.com/tgres/tgres/etc for a sample config file. Make a copy of this file and edit it, at the very least check the db-connect-string setting. Also check log-file directory, it must be writable.

The user of the PostgreSQL database needs CREATE TABLE permissions. On first run tgres will create three tables (ds, rra and ts) and two views (tv and tvd).

Tgres is invoked like this:

$ $GOPATH/bin/tgres -c /path/to/config

For Developers

There is nothing specific you need to know. If you'd like to submit a bug fix, or for anything else - use Github.

Migrating Graphite Data

Included in cmd/whisper_import is a program that can copy whisper data into Tgres, its command-line arguments are self-explanatory. You should be able to start sending data to Tgres and then migrate your Graphite data retroactively by running whisper_import to avoid gaps in data. It's probably a good idea to test a small subset of series first, migrations can be time consuming and resource-intensive.

Owner
tgres
tgres
https://github.com/tgres/tgres
Comments
  • averageSeries with wildcard causes 'invalid memory address or nil pointer dereference'

    averageSeries with wildcard causes 'invalid memory address or nil pointer dereference'

    We have two metrics:

    local.test.a
local.test.b

    We are producing some random data points for these metrics. When we are trying to render one of them, everything looks fine:

    $ curl -s "http://localhost:8888/render?target=averageSeries(local.test.a)&from=$( date +%s --date=-5min )&maxDataPoints=10&format=json"
[
{"target": "averageSeries(local.test.a)", "datapoints": [
[null, 1482321120],[4.733333333333334, 1482321180],[5.383333333333332, 1482321240],[4.783333333333332, 1482321300],[4.1833333333333345, 1482321360],[5.216666666666667, 1482321420]]}]

    But if we try to have average across both metrics:

    $ curl -s "http://localhost:8888/render?target=averageSeries(local.test.*)&from=$( date +%s --date=-5min )&maxDataPoints=10&format=json"

    then we are getting an error and the following stack trace in the logs:

    [9] 2016/12/21 11:51:26 http: panic serving 172.17.0.1:54804: runtime error: invalid memory address or nil pointer dereference
goroutine 14 [running]:
net/http.(*conn).serve.func1(0xc820092a80)
	/usr/lib/go-1.6/src/net/http/server.go:1389 +0xc1
panic(0xaedd80, 0xc820010080)
	/usr/lib/go-1.6/src/runtime/panic.go:443 +0x4e9
database/sql.(*Rows).Close(0x0, 0x0, 0x0)
	/usr/lib/go-1.6/src/database/sql/sql.go:1863 +0x28
github.com/tgres/tgres/serde.(*dbSeries).Close(0xc820109ba0, 0x0, 0x0)
	/usr/local/src/github.com/tgres/tgres/serde/dbseries.go:190 +0x34
github.com/tgres/tgres/dsl.(*SeriesList).Close(0xc820388600, 0x0, 0x0)
	/usr/local/src/github.com/tgres/tgres/dsl/funcs.go:560 +0xa6
github.com/tgres/tgres/http.GraphiteRenderHandler.func1(0x7f849dfd1f70, 0xc820109520, 0xc8201180e0)
	/usr/local/src/github.com/tgres/tgres/http/graphite.go:122 +0x1243
net/http.HandlerFunc.ServeHTTP(0xc820232010, 0x7f849dfd1f70, 0xc820109520, 0xc8201180e0)
	/usr/lib/go-1.6/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc82000b710, 0x7f849dfd1f70, 0xc820109520, 0xc8201180e0)
	/usr/lib/go-1.6/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc820236000, 0x7f849dfd1f70, 0xc820109520, 0xc8201180e0)
	/usr/lib/go-1.6/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc820092a80)
	/usr/lib/go-1.6/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
	/usr/lib/go-1.6/src/net/http/server.go:2137 +0x44e

    Could it be that in dbseries.go:190 we are trying to call Close() on a nil pointer?

  • Add foreground option for Docker use

    Add foreground option for Docker use

    I'd like to use this tool, and I'm happy to setup a Dockerfile for it.

    If you want to add a branch where you are doing the WIP receiver refactor, I can contribute on that also. Or I can wait until you're at a milestone.

  • The gap / backfill problem.

    The gap / backfill problem.

    If we have a series that has not received a data point for a while (i.e. a number of steps), the next data point will cause RRD to fill the gap. This will in turn cause a lot of rows to get updated in the database. If there are many such "stale" series suddenly brought back to life, it causes the vcache to temporarily grow very large followed by a lot of db activity. On a busy set up it is sufficient to leave Tgres stopped for an hour or two to experience this, and there is no easy work-around, it's possible to end up in a situation when Tgres just cannot be started back up. As a band-aid we could impose a limit on the vcache size, but that doesn't really solve the problem.

    One potential way to solve this problem is to not back-fill if the gap exceeds a certain size. The problem is that due to the round-robin nature of our storage, we cannot simply leave slots untouched because they may have previous data in them ("shadow data"). The way to address the sub-problem of shadow data is to add some kind of a version to each data point. Whisper, for example, stores the timestamp, and so if a data point timestamp is outside the range expected, it can be ignored as "expired".

    Adding a timestamp to seems like a lot of extra data for not much gain, but alternatively it could be a 1 byte version. 1 byte would prevent shadow data across 256 iterations of the db, which seems sufficient. If an RRA has been stale for more than 256 times its span, we could have a special rule that prevents its update (i.e. you have to delete and re-create it, for example).

    This would be another backwards-incompatible db schema change, alas.

    Edit: Smallest PG type appears to be SMALLINT which is 2 bytes.

  • Failed to get final advertise address

    Failed to get final advertise address

    Using the default config (except a different db connection string), I get the following error:

    Error initializing cluster, exiting: Failed to get final advertise address: %!v(MISSING)

    To fix this, I set the TGRES_BIND environment variable (to "0.0.0.0" to match the config), but I had to read the source to find it and I'm not sure it's the correct fix.

  • fatal error: concurrent map iteration and map write

    fatal error: concurrent map iteration and map write

    Hi, I installed the latest tgres, and postgres. here's my config:

    
# This is a TOML file: https://github.com/toml-lang/toml

min-step                = "10s"

# 0 - unlilimited (default). points in excess are discarded
max-receiver-queue-size  = 1000000000

# number of flushers == number of workers
workers                 = 4

pid-file =                 "tgres.pid"
log-file =                 "log/tgres.log"
log-cycle-interval =       "24h"

http-listen-spec            = "0.0.0.0:8888"
graphite-line-listen-spec   = "0.0.0.0:2003"
graphite-text-listen-spec   = "0.0.0.0:2003"
graphite-udp-listen-spec    = "0.0.0.0:2003"
graphite-pickle-listen-spec = "0.0.0.0:2004"

statsd-text-listen-spec     = "0.0.0.0:8125"
statsd-udp-listen-spec      = "0.0.0.0:8125"
stat-flush-interval         = "10s"
stats-name-prefix           = "stats"

# RedHat and some others:
#db-connect-string = "host=/tmp dbname=tgres sslmode=disable"
# Debian and some others:
db-connect-string = "host=/var/run/postgresql dbname=tgres sslmode=disable"

[[ds]]
regexp = ".*"
step = "10s"
heartbeat = "2h"
# rra is "[wmean|min|max|last:]ts:ts[:xff]"
# function is not case-sensitive, default is "wmean".
rras = ["10s:6h", "1m:24h", "10m:93d", "1d:5y:1"]

    I created a postgres role and ran createdb tgres, then was able to launch tgres. i started to write some fake data into port 2003 and loaded up http://localhost:8888/ in my browser. but then tgres aborted, as you can see in the output below.

    tgres -c tgres.conf
[16639] 2017/03/12 22:26:18 Tgres starting.
[16639] 2017/03/12 22:26:18 Using config file: 'tgres.conf'.
[16639] 2017/03/12 22:26:18 Logs will be written to '/home/dieter/go/src/github.com/tgres/tgres/etc/log/tgres.log'.
[16639] 2017/03/12 22:26:18 Will cycle logs every 24h0m0s (log-cycle-interval).
[16639] 2017/03/12 22:26:18 All further status messages will be written to log file(s) in '/home/dieter/go/src/github.com/tgres/tgres/etc/log/'.
Graphite text protocol Listening on 0.0.0.0:2003
Graphite UDP protocol Listening on 0.0.0.0:2003
Graphite Pickle protocol Listening on 0.0.0.0:2004
Statsd UDP protocol Listening on 0.0.0.0:8125
HTTP protocol Listening on 0.0.0.0:8888
fatal error: concurrent map iteration and map write

goroutine 65 [running]:
runtime.throw(0x99daef, 0x26)
	/usr/lib/go/src/runtime/panic.go:596 +0x95 fp=0xc420384bf8 sp=0xc420384bd8
runtime.mapiternext(0xc420384d98)
	/usr/lib/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc420384ca8 sp=0xc420384bf8
github.com/tgres/tgres/receiver.(*verticalCache).update(0xc4201f6380, 0xbf0dc0, 0xc42013c040)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/vcache.go:93 +0x2af fp=0xc420384e08 sp=0xc420384ca8
github.com/tgres/tgres/receiver.(*dsFlusher).verticalFlush(0xc4200191d0, 0xbf02c0, 0xc420378ed0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:90 +0x9a fp=0xc420384e58 sp=0xc420384e08
github.com/tgres/tgres/receiver.(*dsFlusher).flushToVCache(0xc4200191d0, 0xbf02c0, 0xc420378ed0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:108 +0x5b fp=0xc420384e80 sp=0xc420384e58
github.com/tgres/tgres/receiver.glob..func8(0xc42026c360, 0xbeddc0, 0xc4200191d0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:76 +0x197 fp=0xc420384f18 sp=0xc420384e80
github.com/tgres/tgres/receiver.glob..func13(0xc4201f0500, 0xc420067740, 0xbeddc0, 0xc4200191d0, 0xbe7780, 0xc42000b100, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:351 +0x167 fp=0xc420384fa8 sp=0xc420384f18
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420384fb0 sp=0xc420384fa8
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:245 +0x384

goroutine 1 [chan receive]:
github.com/tgres/tgres/daemon.glob..func11(0xc42000b100, 0xc42015fe40, 0x7ffe3bb06262, 0xa, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/daemon.go:131 +0x133
github.com/tgres/tgres/daemon.Init(0x7ffe3bb06262, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400000)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/daemon.go:249 +0x1156
main.main()
	/home/dieter/go/src/github.com/tgres/tgres/main.go:70 +0x3d1

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
	/usr/lib/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
	/usr/lib/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/lib/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 19 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.glob..func11.1(0xbe7780, 0xc42000b100, 0xc4200676e0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:174 +0x2a
created by github.com/tgres/tgres/receiver.glob..func11
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:177 +0x5d

goroutine 10 [chan receive]:
github.com/tgres/tgres/daemon.glob..func17.1(0xc420168180, 0x3c)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/log.go:71 +0x4c
created by github.com/tgres/tgres/daemon.glob..func17
	/home/dieter/go/src/github.com/tgres/tgres/daemon/log.go:79 +0x6c

goroutine 11 [sleep]:
time.Sleep(0x4e94914f0000)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/daemon.glob..func17.2(0x4e94914f0000)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/log.go:83 +0x2b
created by github.com/tgres/tgres/daemon.glob..func17
	/home/dieter/go/src/github.com/tgres/tgres/daemon/log.go:89 +0x8e

goroutine 12 [chan receive]:
database/sql.(*DB).connectionOpener(0xc420085180)
	/usr/lib/go/src/database/sql/sql.go:837 +0x4a
created by database/sql.Open
	/usr/lib/go/src/database/sql/sql.go:582 +0x212

goroutine 14 [chan receive]:
github.com/tgres/tgres/graceful.NewListener.func1(0xc42000e160)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:52 +0x5e
created by github.com/tgres/tgres/graceful.NewListener
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:55 +0x108

goroutine 15 [IO wait]:
net.runtime_pollWait(0x7f82c5716118, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005c848, 0x72, 0x0, 0xc4203457c0)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005c848, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005c7e0, 0x0, 0xbe5180, 0xc4203457c0)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e158, 0xc42002b6d0, 0x40e4e8, 0x94a3a0)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e158, 0x1, 0xc42002b738, 0x43372b, 0xc42002b710)
	/usr/lib/go/src/net/tcpsock.go:228 +0x49
github.com/tgres/tgres/graceful.(*Listener).Accept(0xc420167940, 0x9adb00, 0xc42000b100, 0xbee180, 0xc42035ca30)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:68 +0x37
github.com/tgres/tgres/daemon.(*graphiteTextServiceManager).graphiteTextServer(0xc4201678e0, 0x31, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:417 +0x3d
created by github.com/tgres/tgres/daemon.(*graphiteTextServiceManager).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:408 +0x207

goroutine 16 [IO wait]:
net.runtime_pollWait(0x7f82c5716058, 0x72, 0x7)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005c8b8, 0x72, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005c8b8, 0xc4201aa000, 0x1000)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc42005c850, 0xc4201aa000, 0x1000, 0x1000, 0x0, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e168, 0xc4201aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/net.go:181 +0x70
bufio.(*Scanner).Scan(0xc42002bf30, 0xbee0c0)
	/usr/lib/go/src/bufio/scan.go:207 +0x294
github.com/tgres/tgres/daemon.handleGraphiteTextProtocol(0xc42000b100, 0xbee0c0, 0xc42000e168, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:455 +0x105
created by github.com/tgres/tgres/daemon.(*graphiteUdpTextServiceManager).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:357 +0x213

goroutine 34 [chan receive]:
github.com/tgres/tgres/graceful.NewListener.func1(0xc42000e178)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:52 +0x5e
created by github.com/tgres/tgres/graceful.NewListener
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:55 +0x108

goroutine 35 [IO wait]:
net.runtime_pollWait(0x7f82c5715f98, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005c928, 0x72, 0x0, 0xc420120020)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005c928, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005c8c0, 0x0, 0xbe5180, 0xc420120020)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e170, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e170, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock.go:228 +0x49
github.com/tgres/tgres/graceful.(*Listener).Accept(0xc4201679a0, 0x0, 0x0, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:68 +0x37
github.com/tgres/tgres/daemon.(*graphitePickleServiceManager).graphitePickleServer(0xc420167900, 0x1, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:223 +0x3d
created by github.com/tgres/tgres/daemon.(*graphitePickleServiceManager).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:214 +0x1e6

goroutine 36 [IO wait]:
net.runtime_pollWait(0x7f82c5715ed8, 0x72, 0x9)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005c998, 0x72, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005c998, 0xc4201a8000, 0x1000)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc42005c930, 0xc4201a8000, 0x1000, 0x1000, 0x0, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e180, 0xc4201a8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/net.go:181 +0x70
bufio.(*Scanner).Scan(0xc42002d730, 0xbee0c0)
	/usr/lib/go/src/bufio/scan.go:207 +0x294
github.com/tgres/tgres/daemon.handleStatsdTextProtocol(0xc42000b100, 0xbee0c0, 0xc42000e180, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:509 +0x105
created by github.com/tgres/tgres/daemon.(*statsdUdpTextServiceManager).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:576 +0x213

goroutine 37 [chan receive]:
github.com/tgres/tgres/graceful.NewListener.func1(0xc42000e190)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:52 +0x5e
created by github.com/tgres/tgres/graceful.NewListener
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:55 +0x108

goroutine 38 [IO wait]:
net.runtime_pollWait(0x7f82c5715e18, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005ca08, 0x72, 0x0, 0xc420120080)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005ca08, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005c9a0, 0x0, 0xbe5180, 0xc420120080)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e188, 0x429f7f, 0xc42003ce48, 0x410d18)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e188, 0xc420110240, 0xc42003ceb0, 0x4295de, 0xc420110240)
	/usr/lib/go/src/net/tcpsock.go:228 +0x49
github.com/tgres/tgres/graceful.(*Listener).Accept(0xc420167a00, 0xc420110210, 0x8eabe0, 0xc10b50, 0x934320)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:68 +0x37
net/http.(*Server).Serve(0xc4201c2000, 0xbe9d80, 0xc420167a00, 0x0, 0x0)
	/usr/lib/go/src/net/http/server.go:2643 +0x228
github.com/tgres/tgres/daemon.httpServer(0xc42015e370, 0xc, 0xbe9d80, 0xc420167a00, 0xc42000b100, 0xbec860, 0xc4201678c0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/http.go:51 +0x22f
created by github.com/tgres/tgres/daemon.(*wwwServer).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:163 +0x2e1

goroutine 39 [IO wait]:
net.runtime_pollWait(0x7f82c5715d58, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005ca78, 0x72, 0x0, 0xc4201c0000)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005ca78, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005ca10, 0x0, 0xbe5180, 0xc4201c0000)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e1a0, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e1a0, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock.go:215 +0x49
github.com/hashicorp/memberlist.(*Memberlist).tcpListen(0xc42015a2a0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/net.go:180 +0x33
created by github.com/hashicorp/memberlist.newMemberlist
	/home/dieter/go/src/github.com/hashicorp/memberlist/memberlist.go:123 +0x79d

goroutine 40 [IO wait]:
net.runtime_pollWait(0x7f82c5715c98, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005cae8, 0x72, 0x0, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005cae8, 0xc4201ac000, 0x10000)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).readFrom(0xc42005ca80, 0xc4201ac000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_unix.go:277 +0x1cc
net.(*UDPConn).readFrom(0xc42000e1a8, 0xc4201ac000, 0x10000, 0x10000, 0xc42018b6b8, 0x4407cb, 0x10000, 0x8cc520)
	/usr/lib/go/src/net/udpsock_posix.go:47 +0x6a
net.(*UDPConn).ReadFrom(0xc42000e1a8, 0xc4201ac000, 0x10000, 0x10000, 0x10000, 0x10000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/udpsock.go:109 +0x75
github.com/hashicorp/memberlist.(*Memberlist).udpListen(0xc42015a2a0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/net.go:275 +0xc6
created by github.com/hashicorp/memberlist.newMemberlist
	/home/dieter/go/src/github.com/hashicorp/memberlist/memberlist.go:124 +0x7c2

goroutine 41 [select]:
github.com/hashicorp/memberlist.(*Memberlist).udpHandler(0xc42015a2a0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/net.go:361 +0x333
created by github.com/hashicorp/memberlist.newMemberlist
	/home/dieter/go/src/github.com/hashicorp/memberlist/memberlist.go:125 +0x7e7

goroutine 42 [select]:
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc42015a2a0, 0x3b9aca00, 0xc420067080, 0xc420067020, 0xc4201f01a0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:105 +0x196
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:71 +0x3f0

goroutine 43 [select]:
github.com/hashicorp/memberlist.(*Memberlist).pushPullTrigger(0xc42015a2a0, 0xc420067020)
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:132 +0x1ed
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:77 +0x341

goroutine 44 [select]:
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc42015a2a0, 0xbebc200, 0xc4200670e0, 0xc420067020, 0xc4201f01b0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:105 +0x196
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:83 +0x1ed

goroutine 45 [chan receive]:
github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType.func1(0xc42000e1c8, 0xc4200acfd0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:333 +0x65
created by github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:360 +0x113

goroutine 46 [IO wait]:
net.runtime_pollWait(0x7f82c5715bd8, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005d028, 0x72, 0x0, 0xc4201fa020)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005d028, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005cfc0, 0x0, 0xbe5180, 0xc4201fa020)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e1f8, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e1f8, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock.go:228 +0x49
net/rpc.(*Server).Accept(0xc4200157c0, 0xbeaa40, 0xc42000e1f8)
	/usr/lib/go/src/net/rpc/server.go:623 +0x3b
net/rpc.Accept(0xbeaa40, 0xc42000e1f8)
	/usr/lib/go/src/net/rpc/server.go:681 +0x41
github.com/tgres/tgres/cluster.NewClusterBind.func1(0xc4200acfd0)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:156 +0x3e
created by github.com/tgres/tgres/cluster.NewClusterBind
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:158 +0x781

goroutine 66 [chan receive]:
github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType.func1(0xc42011e008, 0xc4200acfd0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:333 +0x65
created by github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:360 +0x113

goroutine 48 [chan receive]:
github.com/tgres/tgres/receiver.glob..func15(0xbeb940, 0xc4201f63c0, 0xbe7800, 0xc420085220, 0xc4200673e0, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:243 +0x1f2
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:60 +0x31e

goroutine 49 [chan receive]:
github.com/tgres/tgres/receiver.glob..func15(0xbeb940, 0xc4201f63e0, 0xbe7800, 0xc420085220, 0xc4200673e0, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:243 +0x1f2
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:60 +0x31e

goroutine 50 [chan receive]:
github.com/tgres/tgres/receiver.glob..func15(0xbeb940, 0xc4201f6400, 0xbe7800, 0xc420085220, 0xc4200673e0, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:243 +0x1f2
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:60 +0x31e

goroutine 51 [chan receive]:
github.com/tgres/tgres/receiver.glob..func15(0xbeb940, 0xc4201f6420, 0xbe7800, 0xc420085220, 0xc4200673e0, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:243 +0x1f2
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:60 +0x31e

goroutine 52 [sleep]:
time.Sleep(0x2540be400)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.glob..func16(0xc4201f6380, 0xc4200673e0, 0xbe7800, 0xc420085220, 0x2540be400, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:304 +0x2f
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:62 +0x39f

goroutine 53 [chan receive]:
github.com/tgres/tgres/receiver.glob..func14(0xbeb940, 0xc4201f6460, 0xbeddc0, 0xc4200191d0, 0xc420067440, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:173 +0x23d
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:67 +0x4f8

goroutine 54 [sleep]:
time.Sleep(0x37e11d600)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.reportTsTableSize(0x7f82c571a458, 0xc420085220, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:325 +0x34
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:71 +0x5c9

goroutine 55 [select]:
github.com/tgres/tgres/receiver.glob..func5(0xbeb940, 0xc4201f64c0, 0xc420066cc0, 0xbee4e0, 0xc4200acfd0, 0x2540be400, 0xc42015ead0, 0x5, 0xbe7780, 0xc42000b100, ...)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:142 +0x603
created by github.com/tgres/tgres/receiver.glob..func29
	/home/dieter/go/src/github.com/tgres/tgres/receiver/startstop.go:142 +0x168

goroutine 67 [chan receive]:
github.com/tgres/tgres/receiver.glob..func1(0x98bcc5, 0x9, 0xc420252060, 0xc420066cc0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:32 +0x7f
created by github.com/tgres/tgres/receiver.glob..func5
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:111 +0x828

goroutine 68 [sleep]:
time.Sleep(0x2540ae60b)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.glob..func2(0x98bcc5, 0x9, 0xc4202520c0, 0x2540be400)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:62 +0x126
created by github.com/tgres/tgres/receiver.glob..func5
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:115 +0x118

goroutine 56 [select]:
github.com/tgres/tgres/receiver.glob..func19(0xbeb940, 0xc4201f6500, 0xc420066d20, 0xbe4540, 0xc42000b100, 0xbe4580, 0xc42000b100, 0x3b9aca00, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/pacedmetric.go:87 +0x73b
created by github.com/tgres/tgres/receiver.glob..func30
	/home/dieter/go/src/github.com/tgres/tgres/receiver/startstop.go:148 +0x15e

goroutine 57 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.glob..func18(0xc42005d030, 0x3b9aca00, 0x991065, 0x11)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/pacedmetric.go:64 +0x4f
created by github.com/tgres/tgres/receiver.glob..func19
	/home/dieter/go/src/github.com/tgres/tgres/receiver/pacedmetric.go:81 +0x16a

goroutine 58 [runnable]:
github.com/tgres/tgres/receiver.glob..func12(0xbeb940, 0xc4201f65a0, 0xc420066c60, 0x4, 0xbee4e0, 0xc4200acfd0, 0xbe7780, 0xc42000b100, 0xc42005c770, 0xbeddc0, ...)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:259 +0xf58
created by github.com/tgres/tgres/receiver.glob..func21
	/home/dieter/go/src/github.com/tgres/tgres/receiver/startstop.go:68 +0x431

goroutine 59 [chan receive]:
github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType.func1(0xc42000e228, 0xc4200acfd0, 0x2)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:333 +0x65
created by github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:360 +0x113

goroutine 60 [chan receive]:
github.com/tgres/tgres/receiver.glob..func6(0xc420067560, 0xc420066c60)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:32 +0x6e
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:224 +0xffe

goroutine 61 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.reportOverrunQueueSize(0xc4201f65c0, 0xbe7780, 0xc42000b100, 0x3b9aca00)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:156 +0x2b
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:229 +0x140

goroutine 62 [select]:
github.com/tgres/tgres/receiver.elasticCh(0xc420066c60, 0xc420067680, 0xc4201f65c0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/elastic_ch.go:56 +0x5ba
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:232 +0x19d

goroutine 63 [chan receive]:
github.com/tgres/tgres/receiver.glob..func11(0xc4200676e0, 0xc420066c60, 0xc42005c770, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:180 +0x93
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:238 +0x214

goroutine 64 [semacquire]:
sync.runtime_SemacquireMutex(0xc420370884)
	/usr/lib/go/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc420370880)
	/usr/lib/go/src/sync/mutex.go:87 +0x9d
github.com/tgres/tgres/receiver.(*cachedDs).processIncoming(0xc42026c360, 0x0, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/dscache.go:175 +0x6b
github.com/tgres/tgres/receiver.glob..func8(0xc42026c360, 0xbeddc0, 0xc4200191d0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:69 +0x40
github.com/tgres/tgres/receiver.glob..func13(0xc4201f0500, 0xc420067740, 0xbeddc0, 0xc4200191d0, 0xbe7780, 0xc42000b100, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:351 +0x167
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:245 +0x384

goroutine 82 [runnable]:
sync.runtime_Semrelease(0xc420370884)
	/usr/lib/go/src/runtime/sema.go:56 +0x3a
sync.(*Mutex).Unlock(0xc420370880)
	/usr/lib/go/src/sync/mutex.go:126 +0x81
github.com/tgres/tgres/receiver.(*cachedDs).processIncoming(0xc42026c360, 0x0, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/dscache.go:189 +0x37a
github.com/tgres/tgres/receiver.glob..func8(0xc42026c360, 0xbeddc0, 0xc4200191d0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:69 +0x40
github.com/tgres/tgres/receiver.glob..func13(0xc4201f0500, 0xc420067740, 0xbeddc0, 0xc4200191d0, 0xbe7780, 0xc42000b100, 0x2)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:351 +0x167
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:245 +0x384

goroutine 83 [semacquire]:
sync.runtime_SemacquireMutex(0xc4203702e4)
	/usr/lib/go/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc4203702e0)
	/usr/lib/go/src/sync/mutex.go:87 +0x9d
github.com/tgres/tgres/receiver.(*verticalCache).update(0xc4201f6380, 0xbf0dc0, 0xc42013c040)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/vcache.go:90 +0x147
github.com/tgres/tgres/receiver.(*dsFlusher).verticalFlush(0xc4200191d0, 0xbf02c0, 0xc420378ed0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:90 +0x9a
github.com/tgres/tgres/receiver.(*dsFlusher).flushToVCache(0xc4200191d0, 0xbf02c0, 0xc420378ed0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:108 +0x5b
github.com/tgres/tgres/receiver.glob..func8(0xc42026c360, 0xbeddc0, 0xc4200191d0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:76 +0x197
github.com/tgres/tgres/receiver.glob..func13(0xc4201f0500, 0xc420067740, 0xbeddc0, 0xc4200191d0, 0xbe7780, 0xc42000b100, 0x3)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:351 +0x167
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:245 +0x384

goroutine 20 [sleep]:
time.Sleep(0x12a05f200)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.reportRuntime(0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/runtime.go:50 +0x34
created by github.com/tgres/tgres/receiver.glob..func21
	/home/dieter/go/src/github.com/tgres/tgres/receiver/startstop.go:72 +0x4a7

goroutine 21 [select, locked to thread]:
runtime.gopark(0x9ae7a0, 0x0, 0x9896b0, 0x6, 0x18, 0x2)
	/usr/lib/go/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc4201caf50, 0x0, 0x18)
	/usr/lib/go/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc4201caf50)
	/usr/lib/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/lib/go/src/runtime/signal_unix.go:434 +0x2dd
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 69 [sleep]:
time.Sleep(0xa7a358200)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/cluster.(*Cluster).Transition.func3.1(0xa7a358200, 0xc42026e000)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:748 +0x2b
created by github.com/tgres/tgres/cluster.(*Cluster).Transition.func3
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:750 +0x1b3

goroutine 391 [runnable]:
fmt.(*ss).SkipSpace(0xc4202145a0)
	/usr/lib/go/src/fmt/scan.go:304 +0x3f
fmt.(*ss).doScanf(0xc4202145a0, 0x98ac11, 0x8, 0xc420389e10, 0x3, 0x3, 0x0, 0x0, 0x0)
	/usr/lib/go/src/fmt/scan.go:1188 +0x354
fmt.Fscanf(0xbe4300, 0xc420243c80, 0x98ac11, 0x8, 0xc420389e10, 0x3, 0x3, 0x8, 0x8c8520, 0xc42018ec01)
	/usr/lib/go/src/fmt/scan.go:143 +0xc5
fmt.Sscanf(0xc420415290, 0x25, 0x98ac11, 0x8, 0xc420389e10, 0x3, 0x3, 0x500, 0xc420415290, 0xc420389de8)
	/usr/lib/go/src/fmt/scan.go:114 +0xad
github.com/tgres/tgres/daemon.parseGraphitePacket(0xc420415290, 0x25, 0x25, 0x52a, 0xc420415290, 0x25, 0xc1f780, 0x4102894000000000, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:484 +0x15d
github.com/tgres/tgres/daemon.handleGraphiteTextProtocol(0xc42000b100, 0xbee180, 0xc42035ca30, 0xa)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:458 +0x15e
created by github.com/tgres/tgres/daemon.(*graphiteTextServiceManager).graphiteTextServer
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:438 +0x214
  • Value Lists not supported

    Value Lists not supported

    It seems that "Value Lists" syntax {foo,bar,...} from http://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards is not supported by tgres.

    When testing with curl (note that curl has its own idea about interpreting curly braces, so we need to escape them):

    $ curl -v 'http://localhost:8888/render?target=local.random.\{a,b\}&maxDataPoints=10&from=-1h'

    I'm getting the following error in the log:

    [8] 2016/12/22 15:12:32 RenderHandler(): Error parsing "group(\"local.random.\"{a,b})": 1:22: missing ',' in argument list

    This seems to be caused by the fact that we use go parser here: https://github.com/tgres/tgres/blob/master/dsl/dsl.go#L45

  • Connecting to Grafana

    Connecting to Grafana

    Hi. This is more a question about documentation.

    I see you've had success using tgres as a data source for Grafana. Did you have to do anything special to get this working?

    I really agree with your premise that time series should be stored in a relational database, so I'm hoping I can get this working.

  • provide a way to delete data sources

    provide a way to delete data sources

    I'm not sure if this is desirable with the way tgres stores data, but it would be helpful to have a command (or postgresql function) to delete a data source (or documentation if this exists already). #

  • undefined: pq.Array golang go version go1.8 darwin/amd64

    undefined: pq.Array golang go version go1.8 darwin/amd64 

    # github.com/tgres/tgres/serde
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:171: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:179: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:221: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:228: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:293: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:302: undefined: pq.Array```
  • HTTP /render endpoint not returning data

    HTTP /render endpoint not returning data

    What is the correct format for querying data from the render endpoint? I am running tgres in Docker with a Postgres instance and I am populating the ts table with while true; do echo -n "carbon.823:$((RANDOM % 100))|c" | nc -w 1 -u 127.0.0.1 8125; done in a separate terminal. I see data in the ts table, but all of my queries at the HTTP /render endpoint returns "SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data". I am (for the most part) formatting queries such that they work with Graphite.

    Example queries: localhost:8888/render?target=carbon.823 localhost:8888/render?target=stats.cabon.823 http://localhost:8888/render/?target=stats.carbon.823&from=-1d&maxDataPoints=100

  • Graceful restart broken

    Graceful restart broken

    This relates to #4 - since implementing a background/daemon option requires a gracefulRestart immediately on start up.

    [1962] 2016/11/02 13:29:16 Got signal: hangup
[1962] 2016/11/02 13:29:16 gracefulRestart(): Beginning graceful restart with sockets: [0xc42015e128 0xc42015e138 0xc42015e140 0xc42015e148 0xc42015e150] and protos: "gt,gu,gp,su,www"
[1962] 2016/11/02 13:29:16 gracefulRestart(): Forked child, waiting to be killed...
[1992] 2016/11/02 13:29:16 Server starting.
[1992] 2016/11/02 13:29:16 Data Sources will be flushed when cached points exceeds 4096 (max-cached-points).
[1992] 2016/11/02 13:29:16 Data Sources will be flushed after (approximately) 5s (max-cache-duration).
[1992] 2016/11/02 13:29:16 A Data Source will be flushed at most once per 1s (min-cache-duration).
[1992] 2016/11/02 13:29:16 Stats (a la statsd) will be flushed every 10s (stat-flush-interval).
[1992] 2016/11/02 13:29:16 Number of workers (and flushers) will be 4.
[1992] 2016/11/02 13:29:16 Pid saved in /home/grisha/golang/src/github.com/tgres/tgres/tgres.pid.
[1992] 2016/11/02 13:29:16 Initialized DB connection.
[1992] 2016/11/02 13:29:16 Error initializing cluster, exiting: Failed to start TCP listener. Err: listen tcp 0.0.0.0:7946: bind: address already in use
[1992] 2016/11/02 13:29:16 main: Waiting for all other goroutines to finish...
[1992] 2016/11/02 13:29:16 main: All goroutines finished, exiting.
  • Viewing newly added metrics requires restart

    Viewing newly added metrics requires restart

    I'm sending graphite formatted metrics to tgres from a script to test it out, and the /render endpoint returns an empty set for any metrics added after tgres was started.

    To resolve I can force tgres to restart, and then the data is available. Note that all the points that were sent are visible, not just those that were created after the restart. You can see what I mean in the following grafana screenshots:

    Before restart screen shot 2017-09-06 at 7 11 26 am

    After tgres restart at ~07:12 screen shot 2017-09-06 at 7 12 07 am

  • Can't build on Windows?

    Can't build on Windows?

    I'm on Windows 10.

    "go get github.com/tgres/tgres"

    Gives me the following output

    daemon.go:232: undefined: syscall.Kill daemon.go:235: undefined: syscall.SIGUSR1 daemon.go:394: undefined: syscall.Kill daemon.go:394: undefined: syscall.SIGUSR1

    A quick Google search leads me to believe that it's using some very unix-specific things that won't work on Windows.

  • isNonNull is incorrect

    isNonNull is incorrect

    isNonNull() should return the same number of series as it was given, with just 0 or 1s. Our current implementation ads them up and returns a single series, but it shouldn't do this.

    (As such, it probably shouldn't be in the "combine" list in Grafana either as it does not combine.)

  • Metric listing is incorrect sometimes

    Metric listing is incorrect sometimes

    foo.*.blah.*.blah doesn't always work (see tgres.*.receiver.workers.*)

    also drop downs should distinctify the list, e.g. tgres.*.receiver.workers.*.channel.* should not list fill_percent 4 times.

Related tags
DevOps Tools go golang time-series grafana graphite statsd postgresql-database
Kubegres is a Kubernetes operator allowing to create a cluster of PostgreSql instances and manage databases replication, failover and backup.

Kubegres is a Kubernetes operator allowing to deploy a cluster of PostgreSql pods with data replication enabled out-of-the box. It brings simplicity w

Dec 30, 2022
Book-API was made using Golang and PostgreSQL with technique CRUD with mux and pq

Book-API CRUD with PostgreSQL Table of contents ?? General info Technologies Blog Setup General info BAPI or Book-API is a Golang REST API made to sho

Feb 18, 2022
PolarDB Stack is a DBaaS implementation for PolarDB-for-Postgres, as an operator creates and manages PolarDB/PostgreSQL clusters running in Kubernetes. It provides re-construct, failover swtich-over, scale up/out, high-available capabilities for each clusters.
 PolarDB Stack is a DBaaS implementation for PolarDB-for-Postgres, as an operator creates and manages PolarDB/PostgreSQL clusters running in Kubernetes. It provides re-construct, failover swtich-over, scale up/out, high-available capabilities for each clusters.

PolarDB Stack开源版生命周期 1 系统概述 PolarDB是阿里云自研的云原生关系型数据库，采用了基于Shared-Storage的存储计算分离架构。数据库由传统的Share-Nothing，转变成了Shared-Storage架构。由原来的N份计算+N份存储，转变成了N份计算+1份存储

Nov 8, 2022
Golang CRUD using database PostgreSQL, adding some fremework like mux and pq.

Golang CRUD with PostgreSQL Table of contents ?? General info Technologies Blog Setup General info GOPOST or Go-Post is a Golang REST API made to show

Nov 27, 2021
Sensu-go-postgres-metrics - The sensu-go-postgres-metrics is a sensu check that collects PostgreSQL metrics

sensu-go-postgres-metrics Table of Contents Overview Known issues Usage examples

Jan 12, 2022
A simple tool to sync your etcd cluster to PostgreSQL in realtime.

etcd-postgresql-syncer A simple tool to sync your etcd cluster to PostgreSQL in realtime. It sets up a watcher on etcd and commits all changes to Post

Jan 20, 2022
Pagodasite - Pagoda site written in go uses echo postgresql redis-cli
 Pagodasite - Pagoda site written in go uses echo postgresql redis-cli

Pagoda: Rapid, easy full-stack web development starter kit in Go Table of Conten

Feb 20, 2022
Simple backup tool for PostgreSQL

pg_back dumps databases from PostgreSQL Description pg_back is a dump tool for PostgreSQL. The goal is to dump all or some databases with globals at o

Dec 25, 2022
🚢 Go package providing lifecycle management for PostgreSQL Docker instances.
 🚢 Go package providing lifecycle management for PostgreSQL Docker instances.

?? psqldocker powered by ory/dockertest. Go package providing lifecycle management for PostgreSQL Docker instances. Leverage Docker to run unit and in

Sep 2, 2022
El Carro is a new project that offers a way to run Oracle databases in Kubernetes as a portable, open source, community driven, no vendor lock-in container orchestration system. El Carro provides a powerful declarative API for comprehensive and consistent configuration and deployment as well as for real-time operations and monitoring.

El Carro: The Oracle Operator for Kubernetes Run Oracle on Kubernetes with El Carro El Carro is a new project that offers a way to run Oracle database

Jan 7, 2023
The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases

The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases. It eliminates the dependency on a human operator or administrator for the majority of database operations.

Dec 14, 2022
Monitor your Website and APIs from your Computer. Get Notified through Slack, E-mail when your server is down or response time is more than expected.
Monitor your Website and APIs from your Computer. Get Notified through Slack, E-mail when your server is down or response time is more than expected.

StatusOK Monitor your Website and APIs from your computer.Get notified through Slack or E-mail when your server is down or response time is more than

Dec 27, 2022
Open Source runtime tool which help to detect malware code execution and run time mis-configuration change on a kubernetes cluster
Open Source runtime tool which help to detect malware code execution and run time mis-configuration change on a kubernetes cluster

Kube-Knark Project Trace your kubernetes runtime !! Kube-Knark is an open source tracer uses pcap & ebpf technology to perform runtime tracing on a de

Sep 19, 2022
StaticBackend is a simple backend server API handling user mgmt, database, storage and real-time component
 StaticBackend is a simple backend server API handling user mgmt, database, storage and real-time component

StaticBackend is a simple backend that handles user management, database, file storage, forms, and real-time experiences via channel/topic-based communication for web and mobile applications.

Jan 7, 2023
Simple Kubernetes real-time dashboard and management.
 Simple Kubernetes real-time dashboard and management.

Skooner - Kubernetes Dashboard We are changing our name from k8dash to Skooner! Please bear with us as we update our documentation and codebase to ref

Dec 28, 2022
A simple CLI and API client for One-Time Secret

OTS Go client otsgo is a simple CLI and API client for One-Time Secret written i

Dec 27, 2021
Nba-simulation - Golang will be simulating nba match and streaming it real time

NBA Simulation golang in-memory To build and run go build ./nbaSimulation To ru

Feb 21, 2022
Tigris is a modern, scalable backend for building real-time websites and apps.

Tigris Data Getting started These instructions will get you through setting up Tigris Data locally as Docker containers. Prerequisites Make sure that

Dec 27, 2022
After approve this contract, you can use the contract to adventure with multiple characters at the same time
 After approve this contract, you can use the contract to adventure with multiple characters at the same time

MultipleRarity 又又又更新了！ MultipleRarity最新版:0x8ACcaa4b940eaFC41b33159027cDBDb4A567d442 注：角色冷却时间不统一时，可以不用管能不能冒险或升级，合约内部加了筛选，但消耗的gas增加了一点点，介意的可以使用常规修复版。 Mu

Nov 19, 2021