Aerospike Client Go

Aerospike Go Client

Aerospike Client Go Build Status Godoc

An Aerospike library for Go.

This library is compatible with Go 1.9+ and supports the following operating systems: Linux, Mac OS X (Windows builds are possible, but untested).

Up-to-date documentation is available in the Godoc.

You can refer to the test files for idiomatic use cases.

Please refer to CHANGELOG.md for release notes, or if you encounter breaking changes.

Notice:

We have released the Go Client v2, with some breaking API changes. Most changes are minor, and can be fixed with relative ease.

The only major issue is that the behavior of the client when a key does not exist has changed.

It used to return no error, but a nil Record.Bins. Now it returns ErrKeyNotFound error.

This is a significant changes, and you should search your code for all instances of Bins == nil and adapt the code accordingly.

Please refer to the CHANGELOG.md for details.

Usage:

The following is a very simple example of CRUD operations in an Aerospike database.

package main

import (
  "fmt"

  aero "github.com/aerospike/aerospike-client-go"
)

// This is only for this example.
// Please handle errors properly.
func panicOnError(err error) {
  if err != nil {
    panic(err)
  }
}

func main() {
  // define a client to connect to
  client, err := aero.NewClient("127.0.0.1", 3000)
  panicOnError(err)

  key, err := aero.NewKey("test", "aerospike", "key")
  panicOnError(err)

  // define some bins with data
  bins := aero.BinMap{
    "bin1": 42,
    "bin2": "An elephant is a mouse with an operating system",
    "bin3": []interface{}{"Go", 2009},
  }

  // write the bins
  err = client.Put(nil, key, bins)
  panicOnError(err)

  // read it back!
  rec, err := client.Get(nil, key)
  panicOnError(err)

  // delete the key, and check if key exists
  existed, err := client.Delete(nil, key)
  panicOnError(err)
  fmt.Printf("Record existed before delete? %v\n", existed)
}

More examples illustrating the use of the API are located in the examples directory.

Details about the API are available in the docs directory.

Prerequisites

Go version v1.12+ is required.

To install the latest stable version of Go, visit http://golang.org/dl/

Aerospike Go client implements the wire protocol, and does not depend on the C client. It is goroutine friendly, and works asynchronously.

Supported operating systems:

  • Major Linux distributions (Ubuntu, Debian, Red Hat)
  • Mac OS X
  • Windows (untested)

Installation:

  1. Install Go 1.9+ and setup your environment as Documented here.
  2. Get the client in your GOPATH : go get github.com/aerospike/aerospike-client-go
  • To update the client library: go get -u github.com/aerospike/aerospike-client-go

Using gopkg.in is also supported: go get -u gopkg.in/aerospike/aerospike-client-go.v1

Some Hints:

  • To run a go program directly: go run <filename.go>
  • to build: go build -o <output> <filename.go>
  • example: go build -o benchmark tools/benchmark/benchmark.go

Performance Tweaking

We are bending all efforts to improve the client's performance. In our reference benchmarks, Go client performs almost as good as the C client.

To read about performance variables, please refer to docs/performance.md

Tests

This library is packaged with a number of tests. Tests require Ginkgo and Gomega library.

Before running the tests, you need to update the dependencies:

$ go get .

To run all the test cases with race detection:

$ ginkgo -r -race

Examples

A variety of example applications are provided in the examples directory.

Tools

A variety of clones of original tools are provided in the tools directory. They show how to use more advanced features of the library to re-implement the same functionality in a more concise way.

Benchmarks

Benchmark utility is provided in the tools/benchmark directory. See the tools/benchmark/README.md for details.

API Documentation

A simple API documentation is available in the docs directory. The latest up-to-date docs can be found in Godoc.

Google App Engine

To build the library for App Engine, build it with the build tag app_engine. Aggregation functionality is not available in this build.

Reflection, and Object API

To make the library both flexible and fast, we had to integrate the reflection API (methods with [Get/Put/...]Object names) tightly in the library. In case you wanted to avoid mixing those API in your app inadvertently, you can use the build tag as_performance to remove those APIs from the build.

License

The Aerospike Go Client is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE.

Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.

Comments
  • DNS lookup issue when used with Kubernetes

    DNS lookup issue when used with Kubernetes

    In Kubernetes service and pods IPs are dynamic and move around when deploying new services/pods. This causes an issue when using the aerospike client.

    In node_validation.go the host name is resolved to the IP address (using net.LookupHost). However when used in Kubernetes where the service IP changes with new deployments this results in an unusable aerospike connection since the retries are all pointing to the old IP.

    Initial start:-

    2016/10/02 00:46:51 Seeding the cluster. Seeds count: 1 2016/10/02 00:46:51 Node Validator has 1 nodes. 2016/10/02 00:46:51 Node BB90459140A4202 partition generation -1 changed to 1 2016/10/02 00:46:51 Tend finished. Live node count: 1 2016/10/02 00:46:51 New cluster initialized and ready to be used...

    After deleting the aerospike service and deploying a new one, the old IP address is still being used to make the connection:- 2016/10/02 00:47:23 Node BB90459140A4202 10.32.0.132:3000 refresh failed: EOF 2016/10/02 00:47:53 Connection to address 10.32.0.132:3000 failed to establish with error: dial tcp 10.32.0.132:3000: i/o timeout 2016/10/02 00:47:53 Node BB90459140A4202 10.32.0.132:3000: dial tcp 10.32.0.132:3000: i/o timeout 2016/10/02 00:47:53 Connection to address 10.32.0.132:3000 failed to establish with error: dial tcp 10.32.0.132:3000: i/o timeout 2016/10/02 00:47:53 Connection to address 10.32.0.132:3000 failed to establish with error: dial tcp 10.32.0.132:3000: i/o timeout 2016/10/02 00:47:53 Connection to address 10.32.0.132:3000 failed to establish with error: dial tcp 10.32.0.132:3000: i/o timeout 2016/10/02 00:47:53 Node BB90459140A4202 10.32.0.132:3000: dial tcp 10.32.0.132:3000: i/o timeout 2016/10/02 00:47:53 Connection to address 10.32.0.132:3000 failed to establish with error: dial tcp 10.32.0.132:3000: i/o timeout

  • Any ideas to warmup connections?

    Any ideas to warmup connections?

    When starting a service we got many errors like:

    last error: No available connections to the node. Connection Pool was empty, and limited to certain number of connections.
    

    We have like 40k qps per instance into Aerospike, so connections are not warmed up at the start. It takes a lot of time to initiate the new ones. Increasing queue length helps a bit.

    So it will be good to warm up conn. queue at the start by setting some options.

  • Growing active connections

    Growing active connections

    Hello! I have a problem. I have a cluster aerospike servers, it consists of 5 nodes. My problem is that the number of active connections is growing and my go service is fall upon reaching a certain number of active connections.

    Driver version -- github.com/aerospike/aerospike-client-go v4.5.2+incompatible Version aerospike server - 5.5.0.3

    Please tell me what could be the problem.

  • cannot use (type string) as type aerospike.Value in argument to lmap.Put

    cannot use (type string) as type aerospike.Value in argument to lmap.Put

    It seems the lmap.Put wont accept my key and value as it is expecting aerospike.Value as type but when i try asserting the type into aerospike.Value, it wont accept it as well. lmap.PutMap works fine

  • Aerospike-go-cilent: High latency and Connections stuck in CLOSE_WAIT

    Aerospike-go-cilent: High latency and Connections stuck in CLOSE_WAIT

    I am trying to run a server written in Golang which internally calls aerospike client (latest version).

    Initially after server start latencies measured from the client side of aerospike were under control but after some time latencies are getting high and not coming down.

    Based on my observation during high latencies, many connections from aerospike client to aerospike server are in close_wait state

    I am using default parameters for client policy i.e, ConnectionQueueSize = 256 and LimitConnectionsToQueueSize = true

    My guess is since many connections are in close_wait state and we are limiting connections it is waiting for connection to close and create a new one and because of that latencies are high

    If that is correct, How can we close the connections faster that are in the close_wait state? Otherwise, What can be the reason for this?

    P.S,

    1. We measured latency on aerospike server (v3.12.1.2) and it was under 1ms (using asloglatency,which is as expected)
    2. No Network congestion from aerospike server side as well as aerospike client side.
  • Put() results in a Invalid node exception after CP update

    Put() results in a Invalid node exception after CP update

    Hi @khaf,

    After https://github.com/aerospike/aerospike-client-go/commit/c4fa10e1a2c8f4d19f57d54c28b9c5fe0dac2fe6 , calling Put() with a client that is directly connected to a single node results in a Invalid node exception. This was working prior to this commit. Get() still works as expected and CP mode is not enabled. Is this expected behavior? Would it be possible to reenable the old behavior? The old behavior is extremely convenient when talking to a cluster in a separate network.

    Thanks, Chiam

  • v1.32.0 results in lots of i/o timeouts

    v1.32.0 results in lots of i/o timeouts

    I updated from v1.31.0 to v1.32.0 and notice a lot of "read tcp 10.20.105.2:47888->10.20.73.2:3000: i/o timeout" errors. Are there any configuration changes needed to migrate from v1.31.0 to v1.32.0?

  • ExecuteUDF OnComplete() returns EOF

    ExecuteUDF OnComplete() returns EOF

    I run a batch job like this:

        res, err := c.ExecuteUDF(
            nil,
            as.NewStatement("my_ns", ""),
            "my_module",
            "my_function",
            as.NewValue(my_first_value),
            as.NewValue(my_second_value),
        )
        if err != nil {
            return err
        }
        return <-res.OnComplete()
    

    This works, the job runs successfully according to AMC, but it returns an EOF error. The job takes about 12 minutes to finish. In a previous version of the Go client the error wasn't there. This is with the latest Go client, and server 3.10.0.3. Any ideas?

  • Using Client.GetObject fails with

    Using Client.GetObject fails with "interface conversion: interface {} is bool, not int"

    I'm trying to use GetObject to marshal the bin data into a struct containing a handful of bool values. The error comes from the lines below in read_command_reflect.go, where value is apparently already a bool, so trying to convert it to an int fails.

    case reflect.Bool:
    	f.SetBool(value.(int) == 1)
    

    I assume this could also happen with the other more complicated case reflect.Bool in the switch under the reflect.Ptr case further down.

    When I replace the first bool case with the following, everything seems to work:

    case reflect.Bool:
    	switch v := value.(type) {
    	case int:
    		f.SetBool(int(v) == 1)
    	default:
    		f.SetBool(bool(value.(bool)))
    	}
    

    I'm not sure if this is an optimal solution, but I'm happy to put up a PR if it seems okay.

  • Need 'omitempty' functionality with 'as' struct tag.

    Need 'omitempty' functionality with 'as' struct tag.

    The issue I am facing is :

    type Category struct {
    	Code string `as:"code"`
    	Description string `as:"description"`
    }
    

    I write a record and populate category Code and Description :

    cat := Category{Code:"NB", Description:"NB is the best"}
    policy := as.NewWritePolicy(0,0)
    err = client.PutObject(policy, someKey, cat)
    

    Now, if I just wish to update Description field of the struct, I donot enter Code value and update the document ( call PutObject with WritePolicy having RecordExistsAction = as.UPDATE)

    But, obviously, my code gets rewritten to empty string. Something like omitempty (like in json struct tag) will fix this.

    Is there a better solution at the moment ?

  • Memory leak

    Memory leak

    After updating aerospike-client-go telegraf started leaking memory in aerospike plugin (influxdata/telegraf#4195). The cause is probably related to changes done in between 1.25.0 and 1.32.0.

  • Reference to issue number - #388 ,

    Reference to issue number - #388 ,

    asKey, err := aerospike.NewKey(asNamespace, set, key)
    listPolicy := aerospike.NewListPolicy(aerospike.ListOrderUnordered, aerospike.ListWriteFlagsAddUnique|aerospike.ListWriteFlagsNoFail)	
    
    ops := []*aerospike.Operation{
    		aerospike.AddOp(aerospike.NewBin(count, 1)),
    		aerospike.AddOp(aerospike.NewBin(points, points)),
    		aerospike.ListAppendWithPolicyOp(listPolicy, ids, id),
    		aerospike.MapIncrementOp(aerospike.DefaultMapPolicy(), mapId, count, 1),
    		aerospike.MapIncrementOp(aerospike.DefaultMapPolicy(), mapId, points, points),
                    aerospike.GetOp()
    	}
    
    wp := as.NewWritePolicy(0, 0)
    wp.RespondPerEachOp = false
    record, err := asClient.Operate(wp, asKey, ops...)
    fmt.Println("result->:", record, ",", err)
    

    // output - error is coming as parameter error, Error is not coming after removing aerospike.GetOp() Can you please help with this?

    As you told write_master: read-all op can't have respond-all-ops flag error is coming in this , but still same error is coming

  • Supporting `QueryPartitions` with non-nil filter (secondary index query)

    Supporting `QueryPartitions` with non-nil filter (secondary index query)

    I see the current v6 version does not support QueryPartitions while the filter is non-nil, while the Java library already supports it (I guess?), is there any plan to support it in the near future?

    Go library: https://github.com/aerospike/aerospike-client-go/blob/v6/client.go#L1027

    Java Library: https://github.com/aerospike/aerospike-client-java/blob/68f41e28ebc8de32c372b366b098579b39b11586/client/src/com/aerospike/client/AerospikeClient.java#L2956

  • multiple cluster nodes is showing cluster timeout

    multiple cluster nodes is showing cluster timeout

    Client side code:

    client, err := aerospike.NewClientWithPolicyAndHost(
        aerospike.NewClientPolicy(),
        aerospike.NewHost("127.0.0.1", 3000),
        aerospike.NewHost("127.0.0.1", 4000),
    )
    

    added aerospike configs as well:

    service {
            user root
            group root
    
            pidfile /var/run/aerospike/asd.pid
            proto-fd-max 15000
    }
    logging {
            file /dev/null {
                    context any info
            }
            console {
                    context any info 
            }
    }
    network {
            service {
                    address any
                    port 3000
                    access-address 172.18.0.3
            }
            heartbeat {
                address any
                mode mesh
                port 3002
                mesh-seed-address-port host.docker.internal 3002 # OK to have local node
                mesh-seed-address-port host.docker.internal 4002
                interval 150
                timeout 10
            }
            fabric {
                address any
                    port 3001
            }
    }
    namespace abcd {
            replication-factor 2
            memory-size 1024M
            default-ttl 0 # 5 days, use 0 to never expire/evict.
            nsup-period 120
            storage-engine memory
            # To use file storage backing, comment out the line above and use the
            # following lines instead.
            storage-engine device {
                    file /opt/aerospike/data/abcd.dat
                    filesize 1G
                    data-in-memory true # Store data in memory in addition to file.
            }
    }
    
  • About ClientPolicy.MinConnectionsPerNode vs server's proto-fd-idle-ms

    About ClientPolicy.MinConnectionsPerNode vs server's proto-fd-idle-ms

    Client

    • Go 1.15.3
    • Aerospike: v4.5.0

    Issue

    Here is the drop-idle-connections logic.

    https://github.com/aerospike/aerospike-client-go/blob/b936c147db0d7ac5f881c3bde9df87e9a6829dfa/connection_heap.go#L237

    For example, if we set ClientPolicy.MinConnectionPerNode to 5, these 5 idle connections may have no chance to check if it is idle or already timeout. Of course, This case is under not busy Aerospike access scenario.

    For these idle conns, if it already exceeds server's proto-fd-idle-ms, Aerospike server may proactively close the conn and client conn hasn't realized that. Then the client will get such error

    Node BB9353C99FXXX 10.y.y.y:3000: write tcp 10.x.x.x:46554->10.y.y.y:3000: write: broken pipe
    

    Question

    Is it possible to fully iterate the conns pool and clean up the idle ones?

  • Client occasionally crashes with index out of bounds when getting a node from a partition

    Client occasionally crashes with index out of bounds when getting a node from a partition

    Occasionally when writing to Aerospike, a service crashes in the Aerospike client when attempting to get a node from the current partition. The length of the array is always 0 when the crash occurs.

    The code here could be modified to ensure it's not getting exceeding the bounds of replicas, but I'm trying to understand why this is happening. Any thoughts?

    Client version: 5.7.0

    Example stack trace (excluding service specific code):

    github.com/aerospike/aerospike-client-go/v5.(*Partition).getSequenceNode(0x46d5d3, 0x61f9685e)
    	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/partition.go:195 +0xfc
    github.com/aerospike/aerospike-client-go/v5.(*Partition).GetNodeWrite(0xc01b79eef8, 0x48c6d7)
    	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/partition.go:165 +0x53
    github.com/aerospike/aerospike-client-go/v5.(*operateCommand).getNode(0xc07677f782a295c3, {0x4e94324576a4, 0x1d0a1c0})
    	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/operate_command.go:43 +0x2d
    github.com/aerospike/aerospike-client-go/v5.(*baseCommand).executeAt(0xc03018ae10, {0x13db778, 0xc03018ae10}, 0xc001fabb90, 0x80, {0xc001fabb90, 0x0, 0x1d0a1c0}, 0x0, 0x0)
    	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/command.go:2058 +0x532
    github.com/aerospike/aerospike-client-go/v5.(*baseCommand).execute(0xc001fabb90, {0x13db778, 0xc03018ae10}, 0x4)
    	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/command.go:1991 +0x8a
    github.com/aerospike/aerospike-client-go/v5.(*operateCommand).Execute(...)
    	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/operate_command.go:60
    github.com/aerospike/aerospike-client-go/v5.(*Client).Operate(0xc0001fc840, 0x0, 0x0, {0xc01cf073e0, 0xfc4500, 0xc01cf13eb0})
    	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/client.go:515 +0x3be
    
Go client library for Pilosa

Go Client for Pilosa Go client for Pilosa high performance distributed index. What's New? See: CHANGELOG Requirements Go 1.12 and higher. Install Down

Dec 3, 2022
Golang client for redislabs' ReJSON module with support for multilple redis clients (redigo, go-redis)

Go-ReJSON - a golang client for ReJSON (a JSON data type for Redis) Go-ReJSON is a Go client for ReJSON Redis Module. ReJSON is a Redis module that im

Dec 25, 2022
redis client implement by golang, inspired by jedis.

godis redis client implement by golang, refers to jedis. this library implements most of redis command, include normal redis command, cluster command,

Dec 6, 2022
Go Memcached client library #golang

About This is a memcache client library for the Go programming language (http://golang.org/). Installing Using go get $ go get github.com/bradfitz/gom

Jan 8, 2023
Neo4j Rest API Client for Go lang

neo4j.go Implementation of client package for communication with Neo4j Rest API. For more information and documentation please read Godoc Neo4j Page s

Nov 9, 2022
Neo4j REST Client in golang

DEPRECATED! Consider these instead: https://github.com/johnnadratowski/golang-neo4j-bolt-driver https://github.com/go-cq/cq Install: If you don't ha

Nov 9, 2022
Neo4j client for Golang

neoism - Neo4j client for Go Package neoism is a Go client library providing access to the Neo4j graph database via its REST API. Status System Status

Dec 30, 2022
Go client for Redis

Redigo Redigo is a Go client for the Redis database. Features A Print-like API with support for all Redis commands. Pipelining, including pipelined tr

Jan 1, 2023
Type-safe Redis client for Golang

Redis client for Golang ❤️ Uptrace.dev - distributed traces, logs, and errors in one place Join Discord to ask questions. Documentation Reference Exam

Jan 1, 2023
Go Redis Client

xredis Built on top of github.com/garyburd/redigo with the idea to simplify creating a Redis client, provide type safe calls and encapsulate the low l

Sep 26, 2022
godis - an old Redis client for Go

godis Implements a few database clients for Redis. There is a stable client and an experimental client, redis and exp, respectively. To use any of the

Apr 16, 2022
Google Go Client and Connectors for Redis

Go-Redis Go Clients and Connectors for Redis. The initial release provides the interface and implementation supporting the (~) full set of current Red

Oct 25, 2022
Redis client library for Go

go-redis go-redis is a Redis client library for the Go programming language. It's built on the skeleton of gomemcache. It is safe to use by multiple g

Nov 8, 2022
Type-safe Redis client for Golang

Redis client for Golang ❤️ Uptrace.dev - distributed traces, logs, and errors in one place Join Discord to ask questions. Documentation Reference Exam

Jan 4, 2023
Redis client Mock Provide mock test for redis query

Redis client Mock Provide mock test for redis query, Compatible with github.com/go-redis/redis/v8 Install Confirm that you are using redis.Client the

Dec 27, 2022
GoBigdis is a persistent database that implements the Redis server protocol. Any Redis client can interface with it and start to use it right away.

GoBigdis GoBigdis is a persistent database that implements the Redis server protocol. Any Redis client can interface with it and start to use it right

Apr 27, 2022
HDFS for Go - This is a native golang client for hdfs.

HDFS for Go This is a native golang client for hdfs. It connects directly to the namenode using the protocol buffers API. It tries to be idiomatic by

Dec 24, 2022
PostgreSQL API Client

PostgreSQL API language search PostgreSQL API functions response: We don't use PostgreSQL in the usual way. We do everything through API functions, wh

May 9, 2022
Isomorphic Go client for PostgREST.

Postgrest GO Golang client for PostgREST. The goal of this library is to make an "ORM-like" restful interface. Documentation Full documentation can be

Dec 5, 2022