A GPU-powered real-time analytics storage and query engine.

Last update: Jan 7, 2023

Comments: 16

AresDB

AresDB is a GPU-powered real-time analytics storage and query engine. It features low query latency, high data freshness and highly efficient in-memory and on disk storage management. Please see AresDB's features, architecture design described in the Uber Engineering Blog.

This repo contains the source code of AresDB and debug UI.

Legal Note

AresDB requires the CUDA Toolkit. Please ensure you read, acknowledge, and accept the CUDA End User License Agreement.

Getting started

To get AresDB:

git clone --recursive https://github.com/uber/aresdb.git $GOPATH/src/github.com/uber/aresdb

NVIDIA Driver and CUDA Setup

AresDB needs NVIDIA driver version >= 390.48 and CUDA version 9.1.

Environment Variables

Run the following to make sure the following environment variables are correctly set:

export PATH=/path/to/cuda/bin:${PATH}
export LD_LIBRARY_PATH=/path/to/cuda/lib64:/path/to/aresdb/lib:${LD_LIBRARY_PATH}
export PKG_CONFIG_PATH=${LD_LIBRARY_PATH}/pkgconfig:${PKG_CONFIG_PATH}

Language Requirements

Building and running AresDB requires:

golang 1.11+
C++ compiler that support c++14
cmake 3.12+
nvcc version 9.1

Configure

Run following commands to generate makefile:

cmake -DQUERY_MODE=DEVICE .

Alternatively, if you want to run the query in CPU mode, run following commands:

cmake -DQUERY_MODE=HOST .

Local Test

AresDB is written in C++ (query engine) and Golang (mem store, disk store and other query components). Because of this, we break testing into two parts:

Test Golang Code

Ginkgo

We use Ginkgo as the test framework for running the Golang unit test and coverage. Install Ginkgo first and run

make test-golang

Test C++ Code

google-test

We use google-test as the test framework to test C++ code. Install google-test and set the environment variable, GTEST_ROOT, to the installed location.

After you have installed properly, run

make test-cuda

Run AresDB Server

The following command will start an AresDB server locally. You can start to query the server using a curl command or swagger page.

make run_server

Run AresDB Docker

Please read the Docker page.

Documentation

Interested in learning more about AresDB? Read the blog post

License

Apache 2.0 License, please see LICENSE for details.

Owner

Uber Open Source

Open Source Software at Uber

https://github.com/uber/aresdb https://eng.uber.com/aresdb/

Comments

Modified the Travis configuration file to enable automated FOSSA scans.

I'm from FOSSA and I'm working with the Uber OSPO to automate license scanning. In addition to the changes in this PR, we may also need to add an API key as an environmental variable in the build environment.

fix docker file error for update to go 1.11

fix docker file error in different go version

errors in https://github.com/uber/aresdb/issues/244

1. different go path in change go version

in this PR

just change the go version from 1.9 to 1.11.4

however the gopath in DockerFile not been changed

already fixed

2. go 1.11+'s rule, we should set GO111MODULE

ENV GO111MODULE=on

already fixed

success output

after that:

root@60eadc19dd6d:~/go/src/github.com/uber/aresdb# make run_server
[ 11%] Built target mem
[100%] Built target algorithm
[100%] Built target lib
[100%] Built target aresd
Scanning dependencies of target run_server
Using config file:  config/ares.yaml
{"level":"info","msg":"Bootstrapping service","config":{"Port":9374,"DebugPort":43202,"RootPath":"ares-root","TotalMemorySize":161061273600,"SchedulerOff":false,"Version":"","Env":"","Distributed":false,"Query":{"DeviceMemoryUtilization":0.95,"DeviceChoosingTimeout":10,"TimezoneTable":{"TableName":"api_cities"},"EnableHashReduction":false},"DiskStore":{"WriteSync":true},"HTTP":{"MaxConnections":300,"ReadTimeOutInSeconds":20,"WriteTimeOutInSeconds":300},"Cluster":{"Enable":false,"ClusterName":"","InstanceName":"60eadc19dd6d"},"Gateway":{"Controller":null},"RedoLogConfig":{"Namespace":"","DiskConfig":{"Disabled":false},"KafkaConfig":{"Enabled":false,"Brokers":null,"TopicSuffix":""}},"InstanceConfig":{"ID":"","Namespace":"","Etcd":{"Zone":"","Env":"","Service":"","CacheDir":"","ETCDClusters":null,"SDConfig":{"InitTimeout":null},"WatchWithRevision":0},"HeartbeatConfig":{"Timeout":0,"Interval":0}}}}
{"level":"info","msg":"Reading schema from local MetaStore ares-root/metastore"}
{"level":"info","msg":"Initialized device manager","utilization":0.949999988079071,"timeout":10}
{"level":"info","msg":"DeviceInfo[0]={DeviceID:0 QueryCount:0 TotalMemory:25576865792 TotalAvailableMemory:24298022912 FreeMemory:24298022912 QueryMemoryUsageMap:map[]}\n"}
{"level":"info","msg":"Bootstrapping device"}
{"level":"info","msg":"Finish bootstrapping device"}
{"level":"info","msg":"Initializing shards from local DiskStore ares-root"}
{"level":"info","msg":"HostMemoryManager: initial preloading done"}
{"level":"info","msg":"Start loading snapshots for all table shards"}
{"level":"info","msg":"Finish loading snapshots for all table shards"}
{"level":"info","msg":"Starting archiving scheduler"}
{"level":"info","msg":"Start replaying redo logs for all table shards"}
{"level":"info","msg":"Finish replaying redo logs for all table shards"}
{"level":"info","msg":"Starting HTTP server on port 9374 with max connection 300"}
{"level":"info","msg":"Starting HTTP server on dbg-port 43202"}

Base docker and examples throws error at data ingest "panic: Missing primary key column"

Describe the bug A clear and concise description of what the bug is. Running basic docker and examples throws error: panic: Missing primary key column

To Reproduce Steps to reproduce the behavior: Follow docker instructions Follow example instructions inside docker container

Expected behavior A clear and concise description of what you expected to happen. Be able to run examples in docker container, so I can get to a place where I can query data via swagger interface. Would also be useful to have a plain description of how to get data ingested via swagger interface as trying to dig through go code inside a docker container to trouble shoot is cumbersome.

Full details============= s@a:$ sudo nvidia-docker run -p 9374:9374/tcp -p 43202:43202/tcp -it aresdb:latest [sudo] password for s: root@020b4c0afe0a:/go/src/github.com/uber/aresdb# make run_server >> ./log/aresdb.log 2>&1 & [1] 15 root@020b4c0afe0a:/go/src/github.com/uber/aresdb# cd examples/ root@020b4c0afe0a:/go/src/github.com/uber/aresdb/examples# go build examples.go root@020b4c0afe0a:/go/src/github.com/uber/aresdb/examples# ./examples tables --dataset 1k_trips table cities created table trips created root@020b4c0afe0a:/go/src/github.com/uber/aresdb/examples# ./examples data --dataset 1k_trips panic: Missing primary key column goroutine 1 [running]: github.com/uber/aresdb/utils.StackError(0x0, 0x0, 0x8f01c9, 0x1a, 0x0, 0x0, 0x0, 0x0) /root/go/src/github.com/uber/aresdb/utils/error.go:61 +0x46e github.com/uber/aresdb/client.checkPrimaryKeys(0xc42033a140, 0xc420316ce0, 0x2, 0x2, 0x0, 0x0) /root/go/src/github.com/uber/aresdb/client/connector.go:241 +0x14b github.com/uber/aresdb/client.(*connector).prepareUpsertBatch(0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420316ce0, 0x2, 0x2, 0xc420018fb0, 0x2, 0x2, 0xc4203a6000, ...) /root/go/src/github.com/uber/aresdb/client/connector.go:223 +0xae github.com/uber/aresdb/client.(*connector).Insert(0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420316ce0, 0x2, 0x2, 0xc4203a6000, 0xa, 0x10, 0x0, ...) /root/go/src/github.com/uber/aresdb/client/connector.go:169 +0x247 main.ingestDataForTable(0xb3b020, 0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420014d00, 0x1a) /root/go/src/github.com/uber/aresdb/examples/examples.go:187 +0x961 main.ingestDataForDataSet() /root/go/src/github.com/uber/aresdb/examples/examples.go:115 +0x370 main.main.func1(0xc4200af900, 0xc420316740, 0x0, 0x2) /root/go/src/github.com/uber/aresdb/examples/examples.go:214 +0x20 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200af900, 0xc4203166e0, 0x2, 0x2, 0xc4200af900, 0xc4203166e0) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:766 +0x2c1 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200af680, 0x8e1260, 0xc4200af7d0, 0xc42014df68) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:852 +0x334 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200af680, 0xc42014df58, 0x3) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:800 +0x2b main.main() /root/go/src/github.com/uber/aresdb/examples/examples.go:239 +0x524

goroutine 1 [running]: main.panicIfErr(0xb3b0a0, 0xc4203193e0) /root/go/src/github.com/uber/aresdb/examples/examples.go:67 +0x4a main.ingestDataForTable(0xb3b020, 0xc42026c9a0, 0xc4200149f0, 0x6, 0xc420014d00, 0x1a) /root/go/src/github.com/uber/aresdb/examples/examples.go:188 +0x986 main.ingestDataForDataSet() /root/go/src/github.com/uber/aresdb/examples/examples.go:115 +0x370 main.main.func1(0xc4200af900, 0xc420316740, 0x0, 0x2) /root/go/src/github.com/uber/aresdb/examples/examples.go:214 +0x20 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200af900, 0xc4203166e0, 0x2, 0x2, 0xc4200af900, 0xc4203166e0) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:766 +0x2c1 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200af680, 0x8e1260, 0xc4200af7d0, 0xc42014df68) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:852 +0x334 github.com/uber/aresdb/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200af680, 0xc42014df58, 0x3) /root/go/src/github.com/uber/aresdb/vendor/github.com/spf13/cobra/command.go:800 +0x2b main.main() /root/go/src/github.com/uber/aresdb/examples/examples.go:239 +0x524 root@020b4c0afe0a:~/go/src/github.com/uber/aresdb/examples#
Add license scan report and status
Your FOSSA integration was successful! Attached in this PR is a badge and license report to track scan status in your README.

Below are docs for integrating FOSSA license checks into your CI:

CircleCI

TravisCI

Jenkins

Other
Broker skeleton
changes included:

minor refactoring to prepare for broker implementation: memstore + aql context

skeleton code for broker

TODOs are added including:

implement BrokerSchemaMutator

implement non agg query executor

implement agg query executor

implement different query options in handler

logging and metrics where needed

Failed to use swagger in docker

Describe the bug Failed to use swagger in docker

To Reproduce

Follow: https://github.com/uber/aresdb/blob/master/docker/README.md
Visit http://my-ip:9374/swagger/ (my-ip is the docker host's ip)

Chrome console output:

Refused to apply style from 'http://my-ip:9374/node_modules/swagger-ui-dist/swagger-ui.css' because its MIME type ('text/plain') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
(index):14 GET http://my-ip:9374/node_modules/swagger-ui-dist/swagger-ui-bundle.js net::ERR_ABORTED 404 (Not Found)
(index):15 GET http://my-ip:9374/node_modules/swagger-ui-dist/swagger-ui-standalone-preset.js net::ERR_ABORTED 404 (Not Found)

broker bug fix: 1. send rewritten query to datanode 2. return empty hll result correctly 3. resultSize is not len(result)
send rewritten query to data node which will rewrite enum string value to number

return empty hll result correctly

resultSize is not len(result)

rewritten dim not passed back
[NLJ] implementation layout
add comments for existing many to one join logic

add TODOs for code change to support NLJ

add query context fields needed for NLJ

compiler changes for matching many to many joins
Suns
Update etcdConfig to support uns

Update etcdConfig loading

Refactor etcd heartbeat registration

Only load aresdb instances from jobAssignment if sinkmode is aresDB

Update GetDataValue if type is not string.

Set sarama publisher manual partition for fact table
support ad-hoc polygons in query
support ad-hoc polygon in a new filter in_polygon(<point_column>, <polygon in WKT format>)

allow exporting polygon as virtual dimension "_geoshape"

docker build error

docker build error in last step RUN make aresd -j

Step 27/27 : RUN make aresd -j
 ---> Running in 2bdfb1ea1d1d
Scanning dependencies of target mem
[  5%] Building C object CMakeFiles/mem.dir/cgoutils/memory/malloc.c.o
[ 11%] Linking C shared library lib/libmem.so
[ 11%] Built target mem
Scanning dependencies of target algorithm
[ 16%] Building CUDA object CMakeFiles/algorithm.dir/query/algorithm.cu.o
[ 22%] Building CUDA object CMakeFiles/algorithm.dir/query/dimension_transform.cu.o
[ 27%] Building CUDA object CMakeFiles/algorithm.dir/query/filter.cu.o
[ 33%] Building CUDA object CMakeFiles/algorithm.dir/query/functor.cu.o
[ 38%] Building CUDA object CMakeFiles/algorithm.dir/query/geo_intersects.cu.o
[ 44%] Building CUDA object CMakeFiles/algorithm.dir/query/hash_lookup.cu.o
[ 50%] Building CUDA object CMakeFiles/algorithm.dir/query/hash_reduction.cu.o
[ 55%] Building CUDA object CMakeFiles/algorithm.dir/query/hll.cu.o
[ 61%] Building CUDA object CMakeFiles/algorithm.dir/query/memory.cu.o
[ 66%] Building CUDA object CMakeFiles/algorithm.dir/query/measure_transform.cu.o
[ 72%] Building CUDA object CMakeFiles/algorithm.dir/query/scratch_space_transform.cu.o
[ 77%] Building CUDA object CMakeFiles/algorithm.dir/query/sort_reduce.cu.o
[ 83%] Building CUDA object CMakeFiles/algorithm.dir/query/transform.cu.o
[ 88%] Building CUDA object CMakeFiles/algorithm.dir/query/utils.cu.o
/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during:
            instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
            instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
            instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(31): here
            instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(364): here

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(128): warning: calling a __host__ function from a __host__ __device__ function is not allowed
          detected during:
            instantiation of "Facade::reference thrust::iterator_core_access::dereference(const Facade &) [with Facade=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>]"
(310): here
            instantiation of "thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::reference thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::operator*() const [with Derived=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, Value=ares::EmptyStruct, System=thrust::host_system_tag, Traversal=thrust::random_access_traversal_tag, Reference=ares::EmptyStruct, Difference=std::ptrdiff_t]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/system/detail/sequential/for_each.h(56): here
            instantiation of "InputIterator thrust::system::detail::sequential::for_each(thrust::system::detail::sequential::execution_policy<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/detail/for_each.inl(44): here
            instantiation of "InputIterator thrust::for_each(const thrust::detail::execution_policy_base<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::ColumnIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(282): here
            instantiation of "void ares::calculateBatchIntersection(GeoShapeBatch, InputIterator, uint32_t *, int, uint32_t, uint32_t *, __nv_bool, cudaStream_t) [with InputIterator=ares::ColumnIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(292): here
            instantiation of "int ares::GeoIntersectionContext::run(uint32_t *, InputIterator) [with InputIterator=ares::ColumnIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(171): here
            instantiation of "int ares::InputVectorBinder<ares::GeoIntersectionContext, 1>::bind(InputIterators...) [with InputIterators=<>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(107): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint64_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(366): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int32_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(370): here

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(128): warning: calling a __host__ function from a __host__ __device__ function is not allowed
          detected during:
            instantiation of "Facade::reference thrust::iterator_core_access::dereference(const Facade &) [with Facade=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>]"
(310): here
            instantiation of "thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::reference thrust::iterator_facade<Derived, Value, System, Traversal, Reference, Difference>::operator*() const [with Derived=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, Value=ares::EmptyStruct, System=thrust::host_system_tag, Traversal=thrust::random_access_traversal_tag, Reference=ares::EmptyStruct, Difference=std::ptrdiff_t]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/system/detail/sequential/for_each.h(56): here
            instantiation of "InputIterator thrust::system::detail::sequential::for_each(thrust::system::detail::sequential::execution_policy<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/detail/for_each.inl(44): here
            instantiation of "InputIterator thrust::for_each(const thrust::detail::execution_policy_base<DerivedPolicy> &, InputIterator, InputIterator, UnaryFunction) [with DerivedPolicy=thrust::system::cpp::detail::par_t, InputIterator=ares::GeoBatchIntersectIterator<ares::RecordIDJoinIterator<GeoPointT>>, UnaryFunction=ares::VoidFunctor]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(282): here
            instantiation of "void ares::calculateBatchIntersection(GeoShapeBatch, InputIterator, uint32_t *, int, uint32_t, uint32_t *, __nv_bool, cudaStream_t) [with InputIterator=ares::RecordIDJoinIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(292): here
            instantiation of "int ares::GeoIntersectionContext::run(uint32_t *, InputIterator) [with InputIterator=ares::RecordIDJoinIterator<GeoPointT>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(205): here
            instantiation of "int ares::InputVectorBinder<ares::GeoIntersectionContext, 1>::bind(InputIterators...) [with InputIterators=<>]"
/root/go/src/github.com/uber/aresdb/query/geo_intersects.cu(107): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during:
            instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
            instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
            instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=uint32_t]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(32): here
            instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int64_t]"
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu(372): here

/root/go/src/github.com/uber/aresdb/query/utils.hpp(114): warning: floating-point value does not fit in required integral type
          detected during:
            instantiation of "Value ares::get_identity_value<Value>(AggregateFunction) [with Value=int64_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(608): here
            instantiation of "ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value *, const uint32_t *, const uint32_t *, AggregateFunction) [with Value=int64_t]"
/root/go/src/github.com/uber/aresdb/query/iterator.hpp(650): here
            instantiation of "ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value *, uint32_t *, uint32_t *, AggregateFunction) [with Value=int64_t]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(34): here
            instantiation of "int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with NInput=1, FunctorType=UnaryFunctorType]"
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu(43): here

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(310): warning: calling a __host__ function("ares::GeoBatchIntersectIterator< ::thrust::permutation_iterator< ::ares::VectorPartyIterator< ::GeoPointT> , unsigned int *> > ::dereference const") from a __host__ __device__ function("thrust::iterator_facade< ::ares::GeoBatchIntersectIterator< ::thrust::permutation_iterator< ::ares::VectorPartyIterator< ::GeoPointT> , unsigned int *> > ,  ::ares::EmptyStruct,  ::thrust::system::cpp::detail::tag,  ::thrust::random_access_traversal_tag,  ::ares::EmptyStruct, long> ::operator * const") is not allowed

/usr/local/cuda/bin/../targets/x86_64-linux/include/thrust/iterator/iterator_facade.h(310): warning: calling a __host__ function("ares::GeoBatchIntersectIterator< ::ares::RecordIDJoinIterator< ::GeoPointT> > ::dereference const") from a __host__ __device__ function("thrust::iterator_facade< ::ares::GeoBatchIntersectIterator< ::ares::RecordIDJoinIterator< ::GeoPointT> > ,  ::ares::EmptyStruct,  ::thrust::system::cpp::detail::tag,  ::thrust::random_access_traversal_tag,  ::ares::EmptyStruct, long> ::operator * const") is not allowed

/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = unsigned int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:364:163:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long unsigned int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:366:178:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:370:148:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long int]':
/root/go/src/github.com/uber/aresdb/query/hash_reduction.cu:372:151:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = int]':
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:31:65:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = unsigned int]':
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = unsigned int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = unsigned int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:32:66:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]
/root/go/src/github.com/uber/aresdb/query/utils.hpp: In instantiation of 'Value ares::get_identity_value(AggregateFunction) [with Value = long int]':
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:608:41:   required from 'ares::MeasureOutputIterator<Value>::MeasureOutputIterator(Value*, const uint32_t*, const uint32_t*, AggregateFunction) [with Value = long int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/iterator.hpp:649:74:   required from 'ares::MeasureOutputIterator<Value> ares::make_measure_output_iterator(Value*, uint32_t*, uint32_t*, AggregateFunction) [with Value = long int; uint32_t = unsigned int]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:34:65:   required from 'int ares::OutputVectorBinder<NInput, FunctorType>::transformMeasureOutput(MeasureOutputVector) const [with int NInput = 1; FunctorType = UnaryFunctorType]'
/root/go/src/github.com/uber/aresdb/query/measure_transform.cu:43:108:   required from here
/root/go/src/github.com/uber/aresdb/query/utils.hpp:114:47: warning: overflow in implicit constant conversion [-Woverflow]


[ 94%] Linking CUDA device code CMakeFiles/algorithm.dir/cmake_device_link.o
[100%] Linking CUDA shared library lib/libalgorithm.so
[100%] Built target algorithm
Scanning dependencies of target lib
[100%] Built target lib
Scanning dependencies of target aresd
make[3]: go: Command not found
CMakeFiles/aresd.dir/build.make:480: recipe for target 'CMakeFiles/aresd' failed
make[3]: *** [CMakeFiles/aresd] Error 127
CMakeFiles/Makefile2:546: recipe for target 'CMakeFiles/aresd.dir/all' failed
make[2]: *** [CMakeFiles/aresd.dir/all] Error 2
CMakeFiles/Makefile2:553: recipe for target 'CMakeFiles/aresd.dir/rule' failed
make[1]: *** [CMakeFiles/aresd.dir/rule] Error 2
Makefile:311: recipe for target 'aresd' failed
make: *** [aresd] Error 2
The command '/bin/sh -c make aresd -j' returned a non-zero code: 2

Status of aresdb
What is the status of this product?

Is this product still being maintained by Uber?

Is this product still in active use at Uber?

Are there any benchmarks anywhere?

Anyone know of an active fork?

The last active commits was back in 2019.
Java Support

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like N/A

Describe alternatives you've considered N/A

Additional context Does AresDB support Java? If yes, could you please share some java examples/instructions?
kaungmyat182804.wordpress
Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

Go to '...'

Click on '....'

Scroll down to '....'

See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]

Browser [e.g. chrome, safari]

Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]

OS: [e.g. iOS8.1]

Browser [e.g. stock browser, safari]

Version [e.g. 22]

Additional context Add any other context about the problem here.
kaungmyat182804.wordpress
Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

Go to '...'

Click on '....'

Scroll down to '....'

See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]

Browser [e.g. chrome, safari]

Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]

OS: [e.g. iOS8.1]

Browser [e.g. stock browser, safari]

Version [e.g. 22]

Additional context Add any other context about the problem here.
Error populating query using the examples
Describe the bug I was following the examples given in aresdb/examples to populate the database with the tables, data and queries. Both the table and data worked fine, but the example for the query resulted in panic: query failed with status code 400. I have attached the full error here: error.log.

To Reproduce Follow the example code given at aresdb/examples:

cd examples

go build examples.go

./examples tables --dataset 1k_trips

./examples data --dataset 1k_trips

./examples query --dataset 1k_trips

Expected behavior Expected the database to respond with the response from the queries.

Desktop (please complete the following information):

OS: Ubuntu 18.04.4 LTS

Browser: Firefox 76.0.1

Docker: v19.03.11

nvidia-docker: v19.03.11

Additional context I am using the docker image of the AresDB.

Scalable datastore for metrics, events, and real-time analytics

InfluxDB InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or

Jan 5, 2023

Scalable datastore for metrics, events, and real-time analytics

InfluxDB InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or

Jan 4, 2023

A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice. A simple in-memory database and table implementation are provided, and you can query any data source you want by implementing a few interfaces.

Dec 27, 2022

Real-time Geospatial and Geofencing

Tile38 is an open source (MIT licensed), in-memory geolocation data store, spatial index, and realtime geofence. It supports a variety of object types

Dec 30, 2022

VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.

NOTICE: This project have moved to fuse-query VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse. Features High Perform

Jan 6, 2023

A high performance NoSQL Database Server powered by Go

LedisDB Ledisdb is a high-performance NoSQL database library and server written in Go. It's similar to Redis but store data in disk. It supports many

Dec 26, 2022

Multitiered file storage API built on Filecoin and IPFS

Powergate Powergate is a multitiered file storage API built on Filecoin and IPFS, and an index builder for Filecoin data. It's designed to be modular

Dec 20, 2022

IceFireDB - Distributed disk storage system based on Raft and RESP protocol.

Distributed disk storage database based on Raft and Redis protocol.

Dec 27, 2022

Secure storage for personal records built to comply with GDPR

Databunker Databunker is a Personally Identifiable Information (PII) Data Storage Service built to Comply with GDPR and CCPA Privacy Requirements. Pro

Jan 8, 2023

moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library. moss stands for "memory-oriented s

Dec 18, 2022

A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

Dec 31, 2022

Key-Value Storage written in Go.

kvs kvs is an in-memory key-value storage written in Go. It has 2 different usage. It can be used as a package by importing it to your code or as a se

Jun 15, 2022

The Prometheus monitoring system and time series database.

Prometheus Visit prometheus.io for the full documentation, examples and guides. Prometheus, a Cloud Native Computing Foundation project, is a systems

Dec 31, 2022

VictoriaMetrics: fast, cost-effective monitoring solution and time series database

VictoriaMetrics VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. It is available in binary release

Jan 8, 2023

LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability.

LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability. LinDB stores all monitoring data of ELEME Inc, there is 88TB incremental writes per day and 2.7PB total raw data.

Jan 1, 2023

TalariaDB is a distributed, highly available, and low latency time-series database for Presto

TalariaDB is a distributed, highly available, and low latency time-series database that stores real-time data. It's built on top of Badger DB.

Nov 16, 2022

Export output from pg_stat_activity and pg_stat_statements from Postgres into a time-series database that supports the Influx Line Protocol (ILP).

pgstat2ilp pgstat2ilp is a command-line program for exporting output from pg_stat_activity and pg_stat_statements (if the extension is installed/enabl

Dec 15, 2021

:handbag: Cache arbitrary data with an expiration time.

cache Cache arbitrary data with an expiration time. Features 0 dependencies About 100 lines of code 100% test coverage Usage // New cache c := cache.N

Jan 5, 2023

Time Series Database based on Cassandra with Prometheus remote read/write support

SquirrelDB SquirrelDB is a scalable high-available timeseries database (TSDB) compatible with Prometheus remote storage. SquirrelDB store data in Cass

Oct 20, 2022

A GPU-powered real-time analytics storage and query engine.

AresDB

Legal Note

Getting started

NVIDIA Driver and CUDA Setup

Environment Variables

Language Requirements

Configure

Local Test

Test Golang Code

Ginkgo

Test C++ Code

google-test

Run AresDB Server

Run AresDB Docker

Documentation

License

Owner

Uber Open Source

Comments

Modified the Travis configuration file to enable automated FOSSA scans.

fix docker file error for update to go 1.11

1. different go path in change go version

2. go 1.11+'s rule, we should set GO111MODULE

success output

Base docker and examples throws error at data ingest "panic: Missing primary key column"

Add license scan report and status

Broker skeleton

Failed to use swagger in docker

broker bug fix: 1. send rewritten query to datanode 2. return empty hll result correctly 3. resultSize is not len(result)

[NLJ] implementation layout

Suns

support ad-hoc polygons in query

docker build error

Status of aresdb

Java Support

kaungmyat182804.wordpress

kaungmyat182804.wordpress

Error populating query using the examples

Related tags

Scalable datastore for metrics, events, and real-time analytics

Scalable datastore for metrics, events, and real-time analytics

A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

Real-time Geospatial and Geofencing

VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.

A high performance NoSQL Database Server powered by Go

Multitiered file storage API built on Filecoin and IPFS

IceFireDB - Distributed disk storage system based on Raft and RESP protocol.

Secure storage for personal records built to comply with GDPR

moss - a simple, fast, ordered, persistable, key-val storage library for golang

A distributed MySQL binlog storage system built on Raft

Key-Value Storage written in Go.

The Prometheus monitoring system and time series database.

VictoriaMetrics: fast, cost-effective monitoring solution and time series database

LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability.

TalariaDB is a distributed, highly available, and low latency time-series database for Presto

Export output from pg_stat_activity and pg_stat_statements from Postgres into a time-series database that supports the Influx Line Protocol (ILP).

:handbag: Cache arbitrary data with an expiration time.

Time Series Database based on Cassandra with Prometheus remote read/write support