Solved: Fix #6 #53 #54 (Config) Introduced: #56

• [x] port old HTTP code
  • [x] only port write code, ignore read for now, since we are still benchmarking write ....
  • http read is ported, don't have plan for grpc yet, the serialize still got trouble
• [x] allow config
• [ ] make GRPC write working
  • [ ] need a grpc client
• [ ] move client library out from pkg/bench
  • make it more close to tsdb-proxy

Partially #47 and Partially #49 (Don't have time to support table layout for now)

• [x] write encoded block #45
• [x] read encoded block #46

Optional

• [ ] (need another issue) change the in memory storage to columnar or compressed columnar format, the row format might be too consuming

• [x] support limit by number of points
• [ ] reporter report data into json file
• [x] use docker volume instead of default COW #51
• [ ] collect container metrics?

Fix #35, Fix #36, Fix #38

Use int64 for timestamp, but it could be different precision (second, millisecond, nanosecond),. Since only second can fit into int32 and modern architecture are 64 bit, we just use int64 for second precision as well.

Also this is a chance for us to implement double point (currently, only int is supported) and fix some existing issues in IntSeries along the way of refactoring.

• [x] Rename TimeNano -> T to avoid confusion
  • [x] IntPoint 0faedc643179ff94915fdcfc6386e0e54d51cf5a
  • [x] DoublePoint
• [x] Change V from int -> int64 05f2b7d37922bd6c47a4297ca522428592f6088e
• [x] Properly decode int and double series when decoding request f0fa155a8317e24cdac2f6855d6fddfa10c53660
  • [x] there could be mixed int and double series, may need to use json.RawMessage and deal it like query
  • [x] or use empty series, like UndeterminedSeries to get the type, now using MetaSeries with json.RawMessage
• [x] IntPoint and IntSeries 0faedc643179ff94915fdcfc6386e0e54d51cf5a (simply use refactor provided by Gogland)
  • [x] common
  • [x] bench
  • [x] storage/memory
  • [x] storage/cassandra
  • [ ] collector
• [x] DoublePoint and DoubleSeries
• [ ] add the sorted flag when ingest series
• [x] use interface for memory Data to support both DoubleSeries and IntSeries
• [ ] remove duplicate when doing merge sort (backtrack?)
• [ ] (optional) float32 instead of float64 because saw in AMS250 slide that GPU speed for float32 and float64 has big difference, may need a benchmark for that

The problem is .... can't find the API endpoint for using HTTP API for push

TODO

• [x] ~~support gRPC in the loader~~ this does not work, it does not have a gRPC server, the best I can find is https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage
• which means we need to find the scrapper code to see how they call the underlying storage layer, seems to be Append method in retrieval/target.go, there is an interface called storage.SampleAppender which is implemented by the local storage

Ref

• https://github.com/prometheus/prometheus/blob/master/storage/remote/client.go#L64 it has gRPC endpoint for pushing data, though I didn't find the default url, I guess there is no url when using gRPC? or they are just using protobuf without using gRPC

Naiveschema + Tags

• [x] cli for creating the schema (though actually not very needed) 7dfc37a83caff17513afee8befd96543bb1285ed
• [ ] current naive schema with tags is WRONG, tags should be in partiton key .... 16b0675a43ad93a20c2c88f63251d96926f7708c
• gocql can handle map properly
• [ ] but you can't filter using map, cassandra think this is a very expensive operation and you must use allow fitlering
• [x] read 60cccf712676f814bf4ec2249dd848c2c47e579a
• [x] write 16b0675a43ad93a20c2c88f63251d96926f7708c
• [x] unify the keyspace to avoid sprintf feb07ab108fdfca1f068963652e977b85ca38aef
• [x] split int and double table feb07ab108fdfca1f068963652e977b85ca38aef
• [ ] use buffer like KairosDB did, write large batch into Cassandra, currently just use the batch size of client

Previously, when using rle on value and raw big endian on time, we got 8M, but now we got 19M, further more, if we double the number of points, the file size triples, it goes to 66M instead of 30～40M

Possible reasons

• old benchmark are using tmpfs, which is different from ssd, but du and ls returns same result, and the write buffer size is default to 4KB

TODO

• better dump util

Current test are just plain golang unit test using assert library, which works perfectly with IDE like Gogland but hard to know what is doing in a glance.

Some candidates

• https://onsi.github.io/ginkgo/

When benchmarking in localhost, got the error, could because the loader and the server is in the same machine, need to containerize xephon-k to confirm this.

Seems related to https://github.com/containous/traefik/issues/157 and https://github.com/containous/traefik/issues/127 Though I am not sure if the MaxIdleConnection per host applies to server, I know it works for client.

Though at start, the motivation is to rewrite KairosDB in Golang, a in memory store become very important for the following reasons

• needed by tsdb-ql which is also the idea comes from
• improve performance, it can be a write through/back cache
• if no compression is added, a in memory store is not very hard to write

TODO

• [x] hash series, name + sorted tags 6c9eb506a49344dc40a83f3649a6b77dc5ce0879
• [x] write series to correct IntSeriesStore 28ac98ab95fc9177852f5e35b4c46e7f2238f690
• [ ] index tags
  • can't figure out a good data structure yet, might use set and do set join?
  • now using inverted index, see #27
• [ ] read
  • [x] read all by name
  • [ ] read using filter by time and tag
• [ ] flush/drop data periodically or by threshold
• [ ] write to disk #32

Ref

• https://github.com/allegro/bigcache

when change index -> indexLength, it update a lot of comment, including the doc for the file format.

• [x] search for indexLength should find most (all?) of them
• [x] search for indexOffset
• [x] search for seriesBlock
• [ ] cassandra.md, known in cassandra backend .... saw it in error, caused by renaming markdown file for benchmark

It's been a while since I worked on xephon ... a lot has changed except the fact I never finish the projects I started ... Anyway, even go.ice is start being maintained, so xephon-k should get revived as well. Also there are more tsdb written in go now.

TODO

• [ ] init log to recap what has been done in xephon-k (good old grad school memories)
• [ ] start playground to test missed things
  • [ ] consistent hashing
  • [ ] serialize inverted index (was building from scratch every time)
  • [ ] compress inverted index
  • [ ] more general data model, druid, piont
  • [ ] compaction
  • [ ] different transport, grpc was never really supported
  • [ ] tcp based line protocol
  • [ ] different compression (existing, RLE, delta RLE are too simple ...)
  • [ ] consider use tree for aggregation
• [ ] clean up old codebase, I think most of them are no longer useful in the new code base beyond reference ...
• [ ] a spec for implementation in other language especially java, c++

Dependencies

• [ ] reboot awesome-tsdb, it's been dead for even longer .... my ‘I will finish it next week' is still there ...
• [ ] go.ice, gommon need to change accordingly, with dep it should be much easier using replace directive

A MVP would be

• [ ] store in memory, inverted index + series
• [ ] WAL (I still don't think WAL is that useful for large volume similar time series data), but adding WAL is much harder than remove it ... so this time, do things in right order
• [ ] compaction
• [ ] use consistent hashing to shard series by meta (tags)

Related https://github.com/libtsdb/libtsdb-go/issues/1 Closed columnar format in memory https://github.com/xephonhq/xephon-k/issues/49 original tracker https://github.com/xephonhq/xephon-k/issues/3

• [ ] review old in memory store
• don't care too much about read performance when filter
• [ ] try to keep it in sync with libtsdb, though having copy in client side between different protocol is not a big issue ...
• [ ] try with large amount of data, maybe larger than 4GB, to see if it still works
  • https://github.com/allegro/bigcache
• [ ] might use custom map implementation, default map is not that good ...

Like Xephon-B, it should make use of go.ice and libstdb-go, might need to adjust go.ice and gommon along the way ...

• old codebase will be kept, we will start a new package called xk which is just another name of pkg, it's better than pkgv2 ...
  • simply ask dep to ignore the pkg package would work I suppose
• [ ] empty cmd for server
• [ ] grpc server
• [ ] http server
• [ ] simple in memory storage

Major issues

• benchmark related code will be split into libstdb-go and xephon-b
• #62 move collector somewhere else, might be benchhub, or gommon
• add tracing on server side to know where is our hot spot
• use go.ice as server framework

TODO

• [x] group and clean up old issues, a new tag pre-benchhub should be added
• [ ] make the cassandra backend work again, it should be very easy without rich tag support
• [ ] recap the column store, it should support read at least
  • [ ] the inverted index can just be kept in memory and rebuilt when start, I don't think there is time to deal with high carnality
  • might take the time and port it to gegecece ...