Vald. A Highly Scalable Distributed Vector Search Engine

Vector Data as a Service

Last update: Dec 29, 2022

Comments: 17

What is Vald?

Vald is a highly scalable distributed fast approximate nearest neighbor dense vector search engine.

Vald is designed and implemented based on Cloud-Native architecture.

It uses the fastest ANN Algorithm NGT to search neighbors.

Vald has automatic vector indexing and index backup, and horizontal scaling which made for searching from billions of feature vector data.

Vald is easy to use, feature-rich and highly customizable as you needed.

Go to Get Started page to try out Vald :)

(If you are interested in ANN benchmarks, please refer to the official website.)

Main Features

Asynchronous Auto Indexing
- Usually the graph requires locking during indexing, which causes stop-the-world. But Vald uses distributed index graphs so it continues to work during indexing.
Customizable Ingress/Egress Filtering
- Vald implements it's own highly customizable Ingress/Egress filter.
- Which can be configured to fit the gRPC interface.
  - Ingress Filter: Ability to Vectorize through filter on request.
  - Egress Filter: rerank or filter the searching result with your own algorithm.
Cloud-native based vector searching engine
- Horizontal scalable on memory and CPU for your demand.
Auto Backup for Index data
- Vald has a feature to store the backup of the index data using MySQL or Cassandra which enables disaster recovery.
Distributed Indexing
- Vald distribute vector index to multiple agents, each agent stores different index.
Index Replication
- Vald stores each index in multiple agents which enables index replicas.
- Automatically rebalance the replica when some Vald agent goes down.
Easy to use
- Vald can be easily installed in a few steps.
Highly customizable
- You can configure the number of vector dimensions, the number of replica and etc.
Multi language supported
- Go, Java, Clojure, Node.js, and Python client library are supported.
- gRPC APIs can be triggered by any programming languages which support gRPC.
- REST API is also supported.

Requirements

Kubernetes 1.17~
AVX2 instructions (required by Vald Agent NGT)

Get Started

Please refer to Get Started.

Installation

Using Helm

helm repo add vald https://vald.vdaas.org/charts
helm install vald-cluster vald/vald

If you use the default values.yaml, the nightly images will be installed.

Docker image tagging policy

nightly ... latest build of master branch
vX.X.X ... released versions
latest ... latest build of release versions
stable ... latest long-term supported version

Using Helm-operator

vald-helm-operator

Example

Write example here

Architecture Overview

Please refer here for more details of the architecture overview in the future.

Development

Before your first commit to this repository, it is strongly recommended to run the commands below.

make init

Components

Component	Docker image
Agent NGT
Agent Sidecar
Discoverer
Gateways
Backup Managers
Compressor
Metas
Index Manager
Helm Operator

Contribution

Please read the contribution guide

Contributors

Thanks goes to these wonderful people (emoji key):

_{Yusuke Kato} 💻 🎨 🚧 📆	_{Rintaro Okamura} 💻 📖 🚧 📦	_{Kosuke Morimoto} 💻 💡 🔧 ⚠️	_{Kiichiro YUKAWA} 📖 🚧 ⚠️ ✅	_datelier 💻 🤔	_{Kevin Diu} 📖 💡 ⚠️ ✅	_{Hiroto Funakoshi} 📖 🔧 ⚠️ ✅
_taisho 🎨 📖 💡	_{Pierre Grimaud} 📖	_{Omer Katz} 📖 ✅

LICENSE

vald released under Apache 2.0 license, refer LICENSE file.

Owner

Vector Data as a Service

Everything becomes Vector

https://github.com/vdaas/vald https://vald.vdaas.org

Comments

create test template for using gotests
Signed-off-by: vankichi [email protected]

Description:

I added make command and testing template for generating missing test files using gotests . This PR is related to #325

Note:

We will update the below test files using our template. Missing test files are below (check filled test file will be generated):

files

[x] cmd/agent/ngt/main_test.go

[ ] cmd/cli/vdctl/main_test.go

[x] cmd/discoverer/k8s/main_test.go

[x] cmd/gateway/vald/main_test.go

[x] cmd/manager/backup/cassandra/main_test.go

[x] cmd/manager/backup/mysql/main_test.go

[x] cmd/manager/compressor/main_test.go

[x] cmd/manager/index/main_test.go

[x] cmd/manager/replication/agent/main_test.go

[x] cmd/manager/replication/controller/main_test.go

[x] cmd/meta/cassandra/main_test.go

[x] cmd/meta/redis/main_test.go

[x] hack/benchmark/internal/assets/dataset_test.go

[x] hack/benchmark/internal/assets/loader_test.go

[ ] hack/benchmark/internal/client/ngtd/grpc/client_test.go

[ ] hack/benchmark/internal/client/ngtd/grpc/option_test.go

[ ] hack/benchmark/internal/client/ngtd/rest/client_test.go

[ ] hack/benchmark/internal/client/ngtd/rest/option_test.go

[x] hack/benchmark/internal/e2e/e2e_test.go

[x] hack/benchmark/internal/e2e/option_test.go

[x] hack/benchmark/internal/e2e/strategy/create_index_option_test.go

[x] hack/benchmark/internal/e2e/strategy/create_index_test.go

[x] hack/benchmark/internal/e2e/strategy/insert_option_test.go

[x] hack/benchmark/internal/e2e/strategy/insert_test.go

[x] hack/benchmark/internal/e2e/strategy/remove_option_test.go

[x] hack/benchmark/internal/e2e/strategy/remove_test.go

[x] hack/benchmark/internal/e2e/strategy/search_option_test.go

[x] hack/benchmark/internal/e2e/strategy/search_test.go

[x] hack/benchmark/internal/e2e/strategy/stream_insert_test.go

[x] hack/benchmark/internal/e2e/strategy/stream_remove_test.go

[x] hack/benchmark/internal/e2e/strategy/stream_search_option_test.go

[x] hack/benchmark/internal/e2e/strategy/stream_search_test.go

[ ] hack/benchmark/internal/starter/agent/ngt/ngt_test.go

[ ] hack/benchmark/internal/starter/agent/ngt/option_test.go

[ ] hack/benchmark/internal/starter/external/ngtd/ngtd_test.go

[ ] hack/benchmark/internal/starter/external/ngtd/option_test.go

[ ] hack/benchmark/internal/starter/gateway/vald/vald_test.go

[ ] hack/license/gen/main_test.go

[ ] hack/swagger/main_test.go

[ ] hack/tools/config/agent/ngt/main_test.go

[ ] hack/tools/config/discoverer/k8s/main_test.go

[ ] hack/tools/config/gateway/vald/main_test.go

[ ] hack/tools/config/manager/backup/mysql/main_test.go

[ ] hack/tools/config/meta/redis/main_test.go

[ ] hack/tools/metrics/main_test.go

[x] internal/cache/cache_test.go

[x] internal/cache/cacher/cacher_test.go

[x] internal/cache/gache/gache_test.go

[x] internal/cache/gache/option_test.go

[x] internal/cache/option_test.go

[x] internal/client/agent/ngt/grpc/client_test.go

[x] internal/client/agent/ngt/grpc/option_test.go

[x] internal/client/agent/ngt/rest/client_test.go

[x] internal/client/agent/ngt/rest/option_test.go

[x] internal/client/compressor/client_test.go

[x] internal/client/compressor/option.go

[x] internal/client/compressor/option_test.go

[x] internal/client/discoverer/discover_test.go

[x] internal/client/discoverer/option_test.go

[x] internal/client/gateway/vald/grpc/client_test.go

[x] internal/client/gateway/vald/grpc/option_test.go

[x] internal/client/gateway/vald/rest/client_test.go

[x] internal/client/gateway/vald/rest/option_test.go

[x] internal/config/backoff_test.go

[x] internal/config/backup_test.go

[x] internal/config/cassandra_test.go

[x] internal/config/client_test.go

[x] internal/config/compress.go

[x] internal/config/compress_test.go

[x] internal/config/config_test.go

[x] internal/config/debug_test.go

[x] internal/config/discoverer_test.go

[x] internal/config/filter_test.go

[x] internal/config/gateway_test.go

[x] internal/config/grpc_test.go

[x] internal/config/index_test.go

[x] internal/config/log_test.go

[x] internal/config/meta_test.go

[x] internal/config/mysql_test.go

[x] internal/config/ngt_test.go

[x] internal/config/observability_test.go

[x] internal/config/redis_test.go

[x] internal/config/server_test.go

[x] internal/config/tcp_test.go

[x] internal/config/tls_test.go

[x] internal/config/transport_test.go

[x] internal/core/converter/tensorflow/option_test.go

[x] internal/core/converter/tensorflow/tensorflow_test.go

[x] internal/db/kvs/redis/option_test.go

[x] internal/db/kvs/redis/redis_test.go

[x] internal/db/nosql/cassandra/cassandra_test.go

[x] internal/db/nosql/cassandra/conviction_test.go

[x] internal/db/nosql/cassandra/option_test.go

[x] internal/db/rdb/mysql/model_test.go

[x] internal/db/rdb/mysql/mysql_test.go

[x] internal/db/rdb/mysql/option_test.go

[x] internal/errgroup/group_test.go

[x] internal/errors/cassandra_test.go

[x] internal/errors/compressor.go

[x] internal/errors/mysql_test.go

[x] internal/errors/redis_test.go

[x] internal/info/info_test.go

[x] internal/k8s/metrics/node/node_test.go

[x] internal/k8s/metrics/node/option_test.go

[x] internal/k8s/metrics/pod/option_test.go

[x] internal/k8s/metrics/pod/pod_test.go

[x] internal/k8s/node/node_test.go

[x] internal/k8s/node/option_test.go

[x] internal/k8s/option_test.go

[x] internal/k8s/pod/option_test.go

[x] internal/k8s/pod/pod_test.go

[x] internal/k8s/reconciler_test.go

[x] internal/log/mock/logger_test.go

[x] internal/log/mock/retry_test.go

[x] internal/net/grpc/metric/client_test.go

[x] internal/net/grpc/metric/server_test.go

[x] internal/net/grpc/proto/proto_test.go

[x] internal/net/grpc/status/status_test.go

[x] internal/net/net_test.go

[x] internal/net/tcp/control_darwin_test.go

[x] internal/net/tcp/control_other_test.go

[x] internal/net/tcp/control_unix_test.go

[x] internal/net/tcp/control_windows_test.go

[x] internal/net/tcp/dialer_test.go

[x] internal/net/tcp/option_dialer_test.go

[x] internal/observability/collector/collector_option_test.go

[x] internal/observability/collector/collector_test.go

[x] internal/observability/exporter/jaeger/jaeger_option_test.go

[x] internal/observability/exporter/jaeger/jaeger_test.go

[x] internal/observability/exporter/prometheus/prometheus_option_test.go

[x] internal/observability/exporter/prometheus/prometheus_test.go

[x] internal/observability/metrics/agent/ngt/ngt_test.go

[x] internal/observability/metrics/manager/compressor/compressor_test.go

[x] internal/observability/metrics/mem/mem_test.go

[x] internal/observability/metrics/metrics_test.go

[x] internal/observability/metrics/runtime/cgo/cgo_test.go

[x] internal/observability/metrics/runtime/goroutine/goroutine_test.go

[x] internal/observability/metrics/version/version_test.go

[x] internal/observability/observability_option_test.go

[x] internal/observability/observability_test.go

[x] internal/observability/trace/status_test.go

[x] internal/observability/trace/trace_option_test.go

[x] internal/observability/trace/trace_test.go

[x] internal/params/option_test.go

[x] internal/params/params_test.go

[x] internal/runner/option_test.go

[x] internal/runner/runner_test.go

[x] internal/singleflight/singleflight_test.go

[x] internal/tls/option_test.go

[x] internal/tls/tls_test.go

[x] internal/worker/queue_test.go

[x] internal/worker/queue_option_test.go

[x] internal/worker/worker_test.go

[x] internal/worker/worker_option_test.go

[x] pkg/agent/ngt/config/config_test.go

[x] pkg/agent/ngt/handler/grpc/handler_test.go

[x] pkg/agent/ngt/handler/grpc/option_test.go

[x] pkg/agent/ngt/handler/rest/handler_test.go

[x] pkg/agent/ngt/handler/rest/option_test.go

[x] pkg/agent/ngt/router/option_test.go

[x] pkg/agent/ngt/router/router_test.go

[x] pkg/agent/ngt/service/kvs/kvs_test.go

[x] pkg/agent/ngt/service/kvs/ou_test.go

[x] pkg/agent/ngt/service/kvs/uo_test.go

[x] pkg/agent/ngt/service/ngt_test.go

[x] pkg/agent/ngt/service/vcaches_test.go

[x] pkg/agent/ngt/usecase/agentd_test.go

[x] pkg/discoverer/k8s/config/config_test.go

[x] pkg/discoverer/k8s/handler/grpc/handler_test.go

[x] pkg/discoverer/k8s/handler/grpc/option_test.go

[x] pkg/discoverer/k8s/handler/rest/handler_test.go

[x] pkg/discoverer/k8s/handler/rest/option_test.go

[x] pkg/discoverer/k8s/router/option_test.go

[x] pkg/discoverer/k8s/router/router_test.go

[x] pkg/discoverer/k8s/service/discover_test.go

[x] pkg/discoverer/k8s/service/nodemap_test.go

[x] pkg/discoverer/k8s/service/nodemetricsmap_test.go

[x] pkg/discoverer/k8s/service/option_test.go

[x] pkg/discoverer/k8s/service/podmetricsmap_test.go

[x] pkg/discoverer/k8s/service/podsmap_test.go

[x] pkg/discoverer/k8s/usecase/discovered_test.go

[x] pkg/gateway/vald/config/config_test.go

[x] pkg/gateway/vald/handler/grpc/checklist_test.go

[x] pkg/gateway/vald/handler/grpc/handler_test.go

[x] pkg/gateway/vald/handler/grpc/option_test.go

[x] pkg/gateway/vald/handler/rest/handler_test.go

[x] pkg/gateway/vald/handler/rest/option_test.go

[x] pkg/gateway/vald/router/option_test.go

[x] pkg/gateway/vald/router/router_test.go

[x] pkg/gateway/vald/service/backup_option_test.go

[x] pkg/gateway/vald/service/backup_test.go

[x] pkg/gateway/vald/service/filter_option_test.go

[x] pkg/gateway/vald/service/filter_test.go

[x] pkg/gateway/vald/service/gateway_option_test.go

[x] pkg/gateway/vald/service/gateway_test.go

[x] pkg/gateway/vald/service/meta_option_test.go

[x] pkg/gateway/vald/service/meta_test.go

[x] pkg/gateway/vald/usecase/vald_test.go

[x] pkg/manager/backup/cassandra/config/config_test.go

[x] pkg/manager/backup/cassandra/handler/grpc/handler_test.go

[x] pkg/manager/backup/cassandra/handler/grpc/option_test.go

[x] pkg/manager/backup/cassandra/handler/rest/handler_test.go

[x] pkg/manager/backup/cassandra/handler/rest/option_test.go

[x] pkg/manager/backup/cassandra/router/option_test.go

[x] pkg/manager/backup/cassandra/router/router_test.go

[x] pkg/manager/backup/cassandra/service/cassandra_test.go

[x] pkg/manager/backup/cassandra/usecase/backupd_test.go

[x] pkg/manager/backup/mysql/config/config_test.go

[x] pkg/manager/backup/mysql/handler/grpc/handler_test.go

[x] pkg/manager/backup/mysql/handler/grpc/option_test.go

[x] pkg/manager/backup/mysql/handler/rest/handler_test.go

[x] pkg/manager/backup/mysql/handler/rest/option_test.go

[x] pkg/manager/backup/mysql/model/model_test.go

[x] pkg/manager/backup/mysql/router/option_test.go

[x] pkg/manager/backup/mysql/router/router_test.go

[x] pkg/manager/backup/mysql/service/mysql_test.go

[x] pkg/manager/backup/mysql/usecase/backupd_test.go

[x] pkg/manager/compressor/config/config.go

[x] pkg/manager/compressor/config/config_test.go

[x] pkg/manager/compressor/handler/grpc/handler.go

[x] pkg/manager/compressor/handler/grpc/handler_test.go

[x] pkg/manager/compressor/handler/grpc/option.go

[x] pkg/manager/compressor/handler/grpc/option_test.go

[x] pkg/manager/compressor/handler/rest/handler_test.go

[x] pkg/manager/compressor/handler/rest/option_test.go

[x] pkg/manager/compressor/router/option_test.go

[x] pkg/manager/compressor/router/router_test.go

[x] pkg/manager/compressor/service/backup_option_test.go

[x] pkg/manager/compressor/service/backup_test.go

[x] pkg/manager/compressor/service/compress.go

[x] pkg/manager/compressor/service/compress_option.go

[x] pkg/manager/compressor/service/compress_option_test.go

[x] pkg/manager/compressor/service/compress_test.go

[x] pkg/manager/compressor/service/registerer.go

[x] pkg/manager/compressor/service/registerer_option.go

[x] pkg/manager/compressor/usecase/compressord.go

[x] pkg/manager/compressor/usecase/compressord_test.go

[x] pkg/manager/index/config/config_test.go

[x] pkg/manager/index/handler/grpc/checklist_test.go

[x] pkg/manager/index/handler/grpc/handler_test.go

[x] pkg/manager/index/handler/grpc/option_test.go

[x] pkg/manager/index/handler/rest/handler_test.go

[x] pkg/manager/index/handler/rest/option_test.go

[x] pkg/manager/index/router/option_test.go

[x] pkg/manager/index/router/router_test.go

[x] pkg/manager/index/service/indexer_test.go

[x] pkg/manager/index/service/indexinfos_test.go

[x] pkg/manager/index/service/option_test.go

[x] pkg/manager/index/usecase/indexer_test.go

[x] pkg/manager/replication/agent/config/config_test.go

[x] pkg/manager/replication/agent/handler/grpc/handler_test.go

[x] pkg/manager/replication/agent/handler/rest/handler_test.go

[x] pkg/manager/replication/agent/handler/rest/option_test.go

[x] pkg/manager/replication/agent/router/option_test.go

[x] pkg/manager/replication/agent/router/router_test.go

[x] pkg/manager/replication/agent/usecase/backupd_test.go

[x] pkg/manager/replication/controller/config/config_test.go

[x] pkg/manager/replication/controller/handler/grpc/handler_test.go

[x] pkg/manager/replication/controller/handler/grpc/option_test.go

[x] pkg/manager/replication/controller/handler/rest/handler_test.go

[x] pkg/manager/replication/controller/handler/rest/option_test.go

[x] pkg/manager/replication/controller/router/option_test.go

[x] pkg/manager/replication/controller/router/router_test.go

[x] pkg/manager/replication/controller/service/discover_test.go

[x] pkg/manager/replication/controller/service/nodemap_test.go

[x] pkg/manager/replication/controller/service/nodemetricsmap_test.go

[x] pkg/manager/replication/controller/service/option_test.go

[x] pkg/manager/replication/controller/service/podmetricsmap_test.go

[x] pkg/manager/replication/controller/service/podsmap_test.go

[x] pkg/manager/replication/controller/usecase/discovered_test.go

[x] pkg/meta/cassandra/config/config_test.go

[x] pkg/meta/cassandra/handler/grpc/handler_test.go

[x] pkg/meta/cassandra/handler/grpc/option_test.go

[x] pkg/meta/cassandra/handler/rest/handler_test.go

[x] pkg/meta/cassandra/handler/rest/option_test.go

[x] pkg/meta/cassandra/router/option_test.go

[x] pkg/meta/cassandra/router/router_test.go

[x] pkg/meta/cassandra/service/cassandra_test.go

[x] pkg/meta/cassandra/usecase/meta_test.go

[x] pkg/meta/redis/config/config_test.go

[x] pkg/meta/redis/handler/grpc/handler_test.go

[x] pkg/meta/redis/handler/grpc/option_test.go

[x] pkg/meta/redis/handler/rest/handler_test.go

[x] pkg/meta/redis/handler/rest/option_test.go

[x] pkg/meta/redis/router/option_test.go

[x] pkg/meta/redis/router/router_test.go/

[x] pkg/meta/redis/service/redis_test.go

[x] pkg/meta/redis/usecase/meta_test.go

Related Issue:

#325

How Has This Been Tested?:

Environment:

Golang Version: 1.14

Docker Version: 19.03.5

Kubernetes Version: 1.17.3

NGT Version: 1.9.1

Types of changes:

[ ] Bug fix [type/bug]

[ ] New feature [type/feature]

[x] Add tests [type/test]

[ ] Security related changes [type/security]

[ ] Add documents [type/documentation]

[ ] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[ ] Have you added an explanation of what your changes do and why you'd like us to include them?

[ ] Have you written new tests for your core changes, as applicable?

[ ] Have you successfully ran tests with your changes locally?

Checklist:

[x] I have read the CONTRIBUTING document.

[ ] I have checked open Pull Requests for the similar feature or fixes?

[ ] I have added tests and benchmarks to cover my changes.

[ ] I have ensured all new and existing tests passed.

[ ] I have commented my code, particularly in hard-to-understand areas

[ ] I have updated the documentation accordingly.
[patch] Test/internal/tcp
Author review required!!!! Maybe performance test required before merging.

Description:

Sorry for this PR become so huge :(

This PR includes the following changes:

enhancement: fetch DNS cache in round robin order instead of random order (which may cause performance problem since it is not balanced)

enhancement: do not lookup DNS when the address passed to cachedDialer() is IP address

bug fix: DNS cache expire hook not executed

bug fix: dialerKeepAlive did not set properly

internal/tcp package test

code refactoring

The test coverage is over 90% in internal/net/tcp package.

When I working on this PR, I found a bug about sharing cache. and I created a PR for this bug fix. https://github.com/vdaas/vald/pull/560

Performance result:

Gateway replicas: 10 Agent replicas: 30 Dataset: random-786-100000 Concurrency: 32 Batch size: 100

Insert mode (100000 random vectors)

Nightly branch (20200713) Average VPS for each run:

5395.421578 6424.425950 6425.912179 6461.443722 5988.220358 6493.199904 6486.264278 6335.389072 6129.457926 6659.673990

Variance

Total Average: 6279.9408957

PR-501 Average VPS for each run:

6217.897276 6673.575530 6976.387795 6890.498418 6487.006260 6585.224161 6734.012306 6966.123523 5937.095263 6751.386072

Variance

Total Average: 6621.9206604 (+ 5.4455 % )

Search Mode

Nightly branch (20200714) Average VPS for each run:

55342.011671 55041.653094 54863.910108 55507.475455 54847.253156 55087.350757 54773.136435 55560.020191 55438.817729 55180.390630

Variance

Total Average: 55164.2019226

PR501 Average VPS for each run:

53006.025749 54954.973641 55533.508719 55152.016128 55107.615004 55149.334190 55255.700455 55688.780070 54430.523454 54487.434100

Variance

Total Average: 54876.591151 (- 0.521372124%)

Search Mode (with 20000 random data & 3 index replica)

Nightly branch (20200715): Average VPS for each run:

54754.772502 54210.456157 55277.095015 54375.751102 54606.793579 54980.820024 55043.223001 54279.938913 54407.296268 54939.336378

Variance

Total Average: 54687.5482939

PR 501: Average VPS for each run:

54635.047079 54744.703738 55256.844790 54916.153493 55260.394869 54986.154799 55572.723950 54315.390605 55549.817220 53304.934344

Variance

Total Average: 54854.2164888 (+ 3%)

Related Issue:

How Has This Been Tested?:

Environment:

Go Version: 1.14.3

Docker Version: 19.03.8

Kubernetes Version: 1.18.2

NGT Version: 1.11.5

Types of changes:

[x] Bug fix [type/bug]

[ ] New feature [type/feature]

[x] Add tests [type/test]

[ ] Security related changes [type/security]

[ ] Add documents [type/documentation]

[x] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[x] Have you added an explanation of what your changes do and why you'd like us to include them?

[x] Have you written new tests for your core changes, as applicable?

[x] Have you successfully ran tests with your changes locally?

Checklist:

[x] I have read the CONTRIBUTING document.

[x] I have checked open Pull Requests for the similar feature or fixes?

[x] I have added tests and benchmarks to cover my changes.

[x] I have ensured all new and existing tests passed.

[x] I have commented my code, particularly in hard-to-understand areas

[ ] I have updated the documentation accordingly.
internal/tls test
Just in case I think we need author review for the tls/tls.go file.

Description:

This PR implements the test case of internal/tls package. The test coverage is 93%. The missing test case is as the follow:

Option will never return error so for example tls.go:48 cannot be covered.

no error return from x509.SystemCertPool() call so tls.go:113 cannot be covered.

It also include the new package implementation called test to get the test data path.

Related Issue:

How Has This Been Tested?:

Environment:

Go Version: 1.14.3

Docker Version: 19.03.8

Kubernetes Version: 1.18.2

NGT Version: 1.11.5

Types of changes:

[ ] Bug fix [type/bug]

[ ] New feature [type/feature]

[x] Add tests [type/test]

[ ] Security related changes [type/security]

[ ] Add documents [type/documentation]

[ ] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[x] Have you added an explanation of what your changes do and why you'd like us to include them?

[x] Have you written new tests for your core changes, as applicable?

[x] Have you successfully ran tests with your changes locally?

Checklist:

[x] I have read the CONTRIBUTING document.

[x] I have checked open Pull Requests for the similar feature or fixes?

[x] I have added tests and benchmarks to cover my changes.

[ ] I have ensured all new and existing tests passed.

[ ] I have commented my code, particularly in hard-to-understand areas

[ ] I have updated the documentation accordingly.

[proposal] Test code of the testing guideline

We suggest the following example test template of the testing guideline. We suggest to use gotests to generate the test code, and we will modify the template of gotests to generate the following test code.

The example implementation is here. https://github.com/vdaas/vald/tree/proposal/tls_test/internal/tls

Please note that:

The args struct will only be generated if target function accept arguments.
The fields struct will only be generated if target struct have fields.
For the option.go template, the comments will be generated with the source code, user need to decide which source code to use.

func Test_person_Hoge4(t *testing.T) {
	type args struct { 
		err error
	}
	type fields struct {
		name string
	}
	type want struct {
		want  string
		want1 string
		err   error
	}
	type test struct {
		name       string
		args       args
		fields     fields
		want       want
		checkFunc  func(want, string, string, error) error
		beforeFunc func(args)
		afterFunc  func(args)
	}
	defaultCheckFunc := func(w want, got string, got1 string, err error) error {
		if !errors.Is(err, w.err) {
			return errors.Errorf("got error = %v, want %v", err, w.err)
		}
		if !reflect.DeepEqual(got, w.want) {
			return errors.Errorf("got = %v, want %v", got, w.want)
		}
		if !reflect.DeepEqual(got1, w.want1) {
			return errors.Errorf("got = %v, want %v", got1, w.want1)
		}
		return nil
	}
	tests := []test{
		// TODO test cases
		{
			name:      "test_case_1",
			args:      args{},
			fields:    fields{},
			want:      want{},
			checkFunc: defaultCheckFunc,
		},
                 // TODO test cases
		func() test {
			return test {
				name:      "test_case_2",
				args:      args{},
				fields:    fields{},
				want:      want{},
				checkFunc: defaultCheckFunc,
			}
		}(),
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if tt.beforeFunc != nil {
				tt.beforeFunc(tt.args)
			}
			if tt.afterFunc != nil {
				defer tt.afterFunc(tt.args)
			}
			if tt.checkFunc == nil {
				tt.checkFunc = defaultCheckFunc
			}
			p := &person{
				name: tt.fields.name,
			}

			got, got1, err := p.Hoge4(tt.args.err)
			if err := tt.checkFunc(tt.want, got, got1, err); err != nil {
				t.Errorf("error = %v", err)
			}

		})
	}
}

Option.go template:

func TestWithAddr(t *testing.T) {
	type T = interface{}
	type args struct {
		addr string
		a    int
		b    int
		c    int
		d    int
	}
	type want struct {
		obj *T
		// Uncomment this line if the option returns an error, otherwise delete it
		// err error
	}
	type test struct {
		name string
		args args
		want want
		// Use the first line if the option returns an error. otherwise use the second line
		// checkFunc  func(want, *T, error) error
		// checkFunc  func(want, *T) error
		beforeFunc func(args)
		afterFunc  func(args)
	}

	// Uncomment this block if the option returns an error, otherwise delete it
	/*
	   defaultCheckFunc := func(w want, obj *T, err error) error {
	       if !errors.Is(err, w.err) {
	           return errors.Errorf("got error = %v, want %v", err, w.err)
	       }
	       if !reflect.DeepEqual(obj, w.obj) {
	           return errors.Errorf("got = %v, want %v", obj, w.obj)
	       }
	       return nil
	   }
	*/

	// Uncomment this block if the option do not returns an error, otherwise delete it
	/*
	   defaultCheckFunc := func(w want, obj *T) error {
	       if !reflect.DeepEqual(obj, w.obj) {
	           return fmt.Errorf("got = %v, want %v", obj, w.c)
	       }
	       return nil
	   }
	*/

	tests := []test{
                 // TODO test cases
		{
			name:      "test_case_1",
			args:      args{},
			want:      want{},
			// checkFunc: defaultCheckFunc,
		},
                 // TODO test cases
		func() test {
			return test {
				name:      "test_case_2",
				args:      args{},
				fields:    fields{},
				want:      want{},
				checkFunc: defaultCheckFunc,
			}
		}(),
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if tt.beforeFunc != nil {
				tt.beforeFunc(tt.args)
			}
			if tt.afterFunc != nil {
				defer tt.afterFunc(tt.args)
			}

			// Uncomment this block if the option returns an error, otherwise delete it
			/*
			   if tt.checkFunc == nil {
			       tt.checkFunc = defaultCheckFunc
			   }

			   got := WithAddr(tt.args.addr, tt.args.a, tt.args.b, tt.args.c, tt.args.d)
			   obj := new(T)
			   if err := tt.checkFunc(tt.want, obj, got(obj)); err != nil {
			       t.Errorf("error = %v", err)
			   }
			*/

			// Uncomment this block if the option do not returns an error, otherwise delete it
			/*
			   if tt.checkFunc == nil {
			       tt.checkFunc = defaultCheckFunc
			   }

			   got := WithAddr(tt.args.addr, tt.args.a, tt.args.b, tt.args.c, tt.args.d)
			   obj := new(T)
			   got(obj)
			   if err := tt.checkFunc(tt.want, obj); err != nil {
			       t.Errorf("error = %v", err)
			   }
			*/
		})
	}
}

WIP [patch] divide gateway
Signed-off-by: kpango [email protected]

WIP

please do not review for now

Description

This PR includes a lot of feature.

Related Issue:

How Has This Been Tested?:

Environment:

Golang Version: 1.14.3

Docker Version: 19.03.8

Kubernetes Version: 1.18.2

NGT Version: 1.11.5

Types of changes:

[ ] Bug fix [type/bug]

[x] New feature [type/feature]

[ ] Add tests [type/test]

[ ] Security related changes [type/security]

[ ] Add documents [type/documentation]

[x] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[x] Have you added an explanation of what your changes do and why you'd like us to include them?

[ ] Have you written new tests for your core changes, as applicable?

[ ] Have you successfully ran tests with your changes locally?

Checklist:

[x] I have read the CONTRIBUTING document.

[x] I have checked open Pull Requests for the similar feature or fixes?

[ ] I have added tests and benchmarks to cover my changes.

[ ] I have ensured all new and existing tests passed.

[ ] I have commented my code, particularly in hard-to-understand areas

[ ] I have updated the documentation accordingly.
♻️ Refactoring compressor
Signed-off-by: Rintaro Okamura [email protected]

Description:

Refactoring manager-compressor.

extract worker loop

add Registerer service that processes requested meta vector and registers it asynchronously

in the PreStop phase, the terminated compressor sends meta_vectors in its queue to other compressor.

Related Issue:

Nothing.

How Has This Been Tested?:

local KinD.

Environment:

Golang Version: 1.14

Docker Version: 19.03.5

Kubernetes Version: 1.17.3

NGT Version: 1.9.1

Types of changes:

[ ] Bug fix [type/bug]

[ ] New feature [type/feature]

[ ] Add tests [type/test]

[ ] Security related changes [type/security]

[ ] Add documents [type/documentation]

[X] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[ ] Have you added an explanation of what your changes do and why you'd like us to include them?

[ ] Have you written new tests for your core changes, as applicable?

[ ] Have you successfully ran tests with your changes locally?

Checklist:

[X] I have read the CONTRIBUTING document.

[X] I have checked open Pull Requests for the similar feature or fixes?

[ ] I have added tests and benchmarks to cover my changes.

[ ] I have ensure all new and existing tests passed.

[ ] I have commented my code, particularly in hard-to-understand areas

[ ] I have updated the documentation accordingly.
Add gRPC interceptor and exporters for Prometheus and Jaeger
Signed-off-by: Rintaro Okamura [email protected]

Description:

Introducing Prometheus and Jaeger exporter using OpenCensus-go.

gRPC interceptor and trace spans are implemented to each components.

deployments of Prometheus, Jaeger, and Grafana (for development use)

two default Grafana dashboards are provided.

Vald Cluster Overview

Vald Agent

Vald Cluster Overview dashboard

Vald Agent dashboard

Related Issue:

Nothing.

How Has This Been Tested?:

In my local KinD cluster. By running

$ make k8s/metrics/prometheus/deploy $ make k8s/metrics/jaeger/deploy $ helm install --values vald/values.yaml \ --set defaults.observability.jaeger.enabled=true \ --set gateway.minReplicas=1 \ --set gateway.hpa.enabled=false \ --set gateway.gateway_config.index_replica=3 \ --set gateway.image.tag=pr-131 \ --set agent.minReplicas=6 \ --set agent.hpa.enabled=false \ --set agent.image.tag=pr-131 \ --set agent.ngt.dimension=6 \ --set discoverer.image.tag=pr-131 \ --set compressor.minReplicas=1 \ --set compressor.hpa.enabled=false \ --set compressor.image.tag=pr-131 \ --set backupManager.minReplicas=1 \ --set backupManager.hpa.enabled=false \ --set backupManager.image.tag=pr-131 \ --set meta.minReplicas=1 \ --set meta.hpa.enabled=false \ --set meta.image.tag=pr-131 \ --generate-name vald

UI ports

Grafana: 3000

Prometheus: 9090

Jaeger: 16686

Environment:

Golang Version: 1.13.8

Docker Version: 19.03.5

Kubernetes Version: 1.16.3

NGT Version: 1.8.4

Types of changes:

[ ] Bug fix [type/bug]

[X] New feature [type/feature]

[ ] Add tests [type/test]

[ ] Security related changes [type/security]

[ ] Add documents [type/documentation]

[ ] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[ ] Have you added an explanation of what your changes do and why you'd like us to include them?

[ ] Have you written new tests for your core changes, as applicable?

[ ] Have you successfully ran tests with your changes locally?

Checklist:

[X] I have read the CONTRIBUTING document.

[X] I have checked open Pull Requests for the similar feature or fixes?

[ ] I have added tests and benchmarks to cover my changes.

[ ] I have ensure all new and existing tests passed.

[ ] I have commented my code, particularly in hard-to-understand areas

[ ] I have updated the documentation accordingly.
bugfix discoverer & add discovery cache & bugfix agent index count & add some feature
Signed-off-by: kpango [email protected]

Description:

in this PR includes 2 bugfixes and 6 features and 3 prototypes

bugfix

discoverer returns the same pointer of grpc payload it makes problem so I fixed about that

agent reports index count using non-atomic counter it makes the wrong value, so I fixed that using atomic variable

agent crashes when observability is nil

feature

discoverer now caches each response

internal Cassandra now supports tls authentication and completely support DC aware routing

grpc client now supports rebalanced reconnecting

grpc client now supports DNS optimized balanced connection pooling

internal/net package now overrides official net package

index manager now supports custom creation pool size

prototype

replication manager controller

replication manager agent

internal compressor client

Related Issue:

How Has This Been Tested?:

Environment:

Golang Version: 1.14

Docker Version: 19.03.6-rc2

Kubernetes Version: 1.17.4

NGT Version: 1.9.1

[ ] Bug fix [type/bug]

[x] New feature [type/feature]

[ ] Add tests [type/test]

[ ] Security related changes [type/security]

[ ] Add documents [type/documentation]

[ ] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[ ] Have you added an explanation of what your changes do and why you'd like us to include them?

[ ] Have you written new tests for your core changes, as applicable?

[ ] Have you successfully ran tests with your changes locally?

Checklist:

[x] I have read the CONTRIBUTING document.

[x] I have checked open Pull Requests for the similar feature or fixes?

[ ] I have added tests and benchmarks to cover my changes.

[ ] I have ensured all new and existing tests passed.

[ ] I have commented my code, particularly in hard-to-understand areas

[ ] I have updated the documentation accordingly.
✨[patch] Implementation of agent-sidecar storage backup logic
Description:

In this PR, a blob storage backup logic of agent-sidecar is implemented. Currently, it only supports S3.

Bugfix:

internal/core/ngt/ngt.go ... remove useless C.frees

Related Issue:

Nothing.

How Has This Been Tested?:

Nothing.

Environment:

Golang Version: 1.14.3

Docker Version: 19.03.8

Kubernetes Version: 1.18.2

NGT Version: 1.11.5

Types of changes:

[ ] Bug fix [type/bug]

[X] New feature [type/feature]

[ ] Add tests [type/test]

[ ] Security related changes [type/security]

[ ] Add documents [type/documentation]

[ ] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[ ] Have you added an explanation of what your changes do and why you'd like us to include them?

[ ] Have you written new tests for your core changes, as applicable?

[ ] Have you successfully ran tests with your changes locally?

Checklist:

[X] I have read the CONTRIBUTING document.

[X] I have checked open Pull Requests for the similar feature or fixes?

[ ] I have added tests and benchmarks to cover my changes.

[ ] I have ensured all new and existing tests passed.

[ ] I have commented my code, particularly in hard-to-understand areas

[ ] I have updated the documentation accordingly.
[WIP] add replication manager and replicator
Signed-off-by: kpango [email protected]

Description:

create replication manager

Related Issue:

How Has This Been Tested?:

Environment:

Golang Version: 1.13.5

Docker Version: 19.03.5

Kubernetes Version: 1.16.3

NGT Version: 1.8.4

Types of changes:

[ ] Bug fix [type/bug]

[x] New feature [type/feature]

[ ] Add tests [type/test]

[ ] Security related changes [type/security]

[ ] Add documents [type/documentation]

[ ] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[ ] Have you added an explanation of what your changes do and why you'd like us to include them?

[ ] Have you written new tests for your core changes, as applicable?

[ ] Have you successfully ran tests with your changes locally?

Checklist:

[x] I have read the CONTRIBUTING document.

[x] I have checked open Pull Requests for the similar feature or fixes?

[ ] I have added tests and benchmarks to cover my changes.

[ ] I have ensure all new and existing tests passed.

[ ] I have commented my code, particularly in hard-to-understand areas

[ ] I have updated the documentation accordingly.
Vald architecture document
#106 # Description: This PR create a Vald Architecture document including the summary of each components in Vald and the data flow explanation.

Remaining task:

Update and Delete flow (another PR)

[ ] Fix image design

Related Issue:

How Has This Been Tested?:

Environment:

Golang Version: 1.14

Docker Version: 19.03.5

Kubernetes Version: 1.17.3

NGT Version: 1.9.1

Types of changes:

[ ] Bug fix [type/bug]

[ ] New feature [type/feature]

[ ] Add tests [type/test]

[ ] Security related changes [type/security]

[x] Add documents [type/documentation]

[ ] Refactoring [type/refactoring]

[ ] Update dependencies [type/dependency]

[ ] Update benchmarks and performances [type/bench]

[ ] Update CI [type/ci]

Changes to Core Features:

[x] Have you added an explanation of what your changes do and why you'd like us to include them?

[x] Have you written new tests for your core changes, as applicable?

[x] Have you successfully ran tests with your changes locally?

Checklist:

[x] I have read the CONTRIBUTING document.

[x] I have checked open Pull Requests for the similar feature or fixes?

[x] I have added tests and benchmarks to cover my changes.

[x] I have ensured all new and existing tests passed.

[x] I have commented my code, particularly in hard-to-understand areas

[x] I have updated the documentation accordingly.
Refactor Insert Upsert Testing
Signed-off-by: kpango [email protected]

Description:

SSIA

Related Issue:

Versions:

Go Version: 1.19.4

Docker Version: 20.10.8

Kubernetes Version: 1.22.0

NGT Version: 2.0.8

Checklist:

[ ] I have read the CONTRIBUTING document and completed our CLA agreement.

[ ] I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:
[WIP]Add persistent volume snapshot
Add persistent volume snapshot

Description:

Related Issue:

Versions:

Go Version: 1.19.4

Docker Version: 20.10.8

Kubernetes Version: 1.22.0

NGT Version: 1.14.8

Checklist:

[x] I have read the CONTRIBUTING document and completed our CLA agreement.

[x] I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:
[WIP] Add query to egress filter.
Description:

Enable to pass query in egress filter and filter gateway.

Related Issue:

Versions:

Go Version: 1.19.4

Docker Version: 20.10.8

Kubernetes Version: 1.22.0

NGT Version: 1.14.8

Checklist:

[ ] I have read the CONTRIBUTING document and completed our CLA agreement.

[ ] I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:
[WIP] Add flush api
Description:

Related Issue:

Versions:

Go Version: 1.19.4

Docker Version: 20.10.8

Kubernetes Version: 1.22.0

NGT Version: 1.14.8

Checklist:

[x] I have read the CONTRIBUTING document and completed our CLA agreement.

[x] I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:
Create observability configuration document
Description:

I have created the observability configuration (v1.7~). It requires OpenTelemetry and OpenTelemtry Protocol for observing.

Related Issue:

Versions:

Go Version: 1.19.2

Docker Version: 20.10.8

Kubernetes Version: 1.22.0

NGT Version: 1.14.8

Checklist:

[x] I have read the CONTRIBUTING document and completed our CLA agreement.

[x] I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:
[WIP] Add mirror gateway component
Signed-off-by: hlts2 [email protected]

Description:

Related Issue:

Versions:

Go Version: 1.19.2

Docker Version: 20.10.8

Kubernetes Version: 1.22.0

NGT Version: 1.14.8

Checklist:

[x] I have read the CONTRIBUTING document and completed our CLA agreement.

[x] I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:

Vald. A Highly Scalable Distributed Vector Search Engine

What is Vald?

Main Features

Requirements

Get Started

Installation

Using Helm

Docker image tagging policy

Using Helm-operator

Example

Architecture Overview

Development

Components

Contribution

Contributors

LICENSE

Owner

Vector Data as a Service

Comments

create test template for using gotests

Description:

Related Issue:

How Has This Been Tested?:

Environment:

Types of changes:

Changes to Core Features:

Checklist:

[patch] Test/internal/tcp

Description:

Insert mode (100000 random vectors)

Search Mode

Search Mode (with 20000 random data & 3 index replica)

Related Issue:

How Has This Been Tested?:

Environment:

Types of changes:

Changes to Core Features:

Checklist:

internal/tls test

Description:

Related Issue:

How Has This Been Tested?:

Environment:

Types of changes:

Changes to Core Features:

Checklist:

[proposal] Test code of the testing guideline

WIP [patch] divide gateway

WIP

please do not review for now

Description

Related Issue:

How Has This Been Tested?:

Environment:

Types of changes:

Changes to Core Features:

Checklist:

♻️ Refactoring compressor

Description:

Related Issue:

How Has This Been Tested?:

Environment:

Types of changes:

Changes to Core Features:

Checklist:

Add gRPC interceptor and exporters for Prometheus and Jaeger

Description:

Related Issue:

How Has This Been Tested?:

Environment:

Types of changes:

Changes to Core Features:

Checklist:

bugfix discoverer & add discovery cache & bugfix agent index count & add some feature

Description:

in this PR includes 2 bugfixes and 6 features and 3 prototypes

bugfix

feature

prototype

Related Issue: