Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

[ ] elastic.v2 (for Elasticsearch 1.x) [x ] elastic.v3 (for Elasticsearch 2.x)

Please describe the expected behavior

NewClient(elastic.SetURL("http://:9200") would correctly generate a new Client object connecting to the node

Please describe the actual behavior

"no ElasticSearch node available"

Any steps to reproduce the behavior?

elastic.NewClient(elastic.SetURL("http://:9200")

@dashaus, I copied it over from #57:

Hi, I have the same problem here:

panic: main: conn db: no Elasticsearch node available

goroutine 1 [running]:
log.Panicf(0x84de50, 0x11, 0xc2080c7e90, 0x1, 0x1)
    /usr/local/go/src/log/log.go:314 +0xd0
main.init·1()
    /Users/emilio/go/src/monoculum/init.go:40 +0x348
main.init()
    /Users/emilio/go/src/monoculum/main.go:334 +0xa4

goroutine 526 [select]:
net/http.(*persistConn).roundTrip(0xc2088ad1e0, 0xc2086a9d50, 0x0, 0x0, 0x0)
20:30:13 app         |  /usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc20806c000, 0xc2086f6000, 0xc20873ff50, 0x0, 0x0)
20:30:13 app         |  /usr/local/go/src/net/http/transport.go:235 +0x558
20:30:13 app         | net/http.send(0xc2086f6000, 0xed4f18, 0xc20806c000, 0x21, 0x0, 
20:30:13 app         | 0x0)
    /usr/local/go/src/net/http/client.go:219
20:30:13 app         |  +0x4fc
net/http.(*Client).send(0xc08b00, 0xc2086f6000, 0x21
20:30:13 app         | , 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:142 +0x15b
20:30:13 app         | net/http.(*Client).doFollowingRedirects(0xc08b00, 0xc2086f6000, 0x97cd00, 0x0, 0x0, 0x0)
20:30:13 app         |  /usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0xc08b00, 0xc2086f6000, 0xc20873fce0, 0x0, 
20:30:13 app         | 0x0)
    /usr/local/go/src/net/http/client.go
20:30:13 app         | :174 +0xa4
github.com/olivere/elastic.(*Client).sniffNode(0xc208659d10, 0xc208569920, 0x15
20:30:13 app         | , 0x0, 0x0, 0x0)
20:30:13 app         |  /Users/emilio/go/src/github.com/olivere/elastic/client.go:543
20:30:13 app         |  +0x16a
20:30:13 app         | 
github.com/olivere/elastic.func·014(0xc208569920, 0x15
20:30:13 app         | )
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x47
20:30:13 app         | created by github.com/olivere/elastic.(*Client).sniff
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x744

goroutine 525 [chan receive]:
20:30:13 app         | database/sql.(*DB).connectionOpener(0xc2086de960)
    /usr/local/go/src/database/sql/sql.go:589 +0x4c
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:452 +0x31c

goroutine 529 [IO wait]:
20:30:13 app         | net.(*pollDesc).Wait(0xc2084fe370, 0x72, 0x0
20:30:13 app         | , 
20:30:13 app         | 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2084fe370, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2084fe310, 0xc208709000, 0x1000, 0x1000, 0x0, 0xed4d48, 0xc2086a9ec8)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20896a800, 0xc208709000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0xef0410, 0xc20896a800, 0xc2088ad238, 0xc208709000, 0x1000, 0x1000, 0xeb7010, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc208569b40, 0xc208709000, 0x1000, 0x1000, 0xc207f6957f, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2088f3c80)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2088f3c80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc2088ad1e0)
    /usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 530 [select]:
net/http.(*persistConn).writeLoop(0xc2088ad1e0)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc

This occurs sometimes... not always...

curl -XGET 127.0.0.1:9200/_nodes/http?pretty=1
{
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "3l_Ing0oSfWu5U63US5kxg" : {
      "name" : "Rattler",
      "transport_address" : "inet[192.168.1.91/192.168.1.91:9300]",
      "host" : "Mac-Emilio",
      "ip" : "192.168.1.91",
      "version" : "1.3.4",
      "build" : "a70f3cc",
      "http_address" : "inet[/192.168.1.91:9200]",
      "http" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0:9200]",
        "publish_address" : "inet[/192.168.1.91:9200]",
        "max_content_length_in_bytes" : 104857600
      }
    }
  }
}

I'm running Elasticsearch v1.4.4 in a Docker container. I kept having trouble getting the client to work properly. I was trying to run the sample in the README (obviously pointing to my Docker container instead of localhost). It was taking ~30 seconds to create the client, and then would fail to create the index with the error: no Elasticsearch node available.

As soon as I set turned off sniffing when creating the client (elastic.SetSniff(false)), everything worked perfectly. It doesn't really bother me that I have to turn sniffing off, but I wanted to put this issue out to see if anyone else had seen an issue like this.

P.S. @olivere - The documentation is awesome! :+1:

Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

[ ] elastic.v5 (for Elasticsearch 5.x)

Please describe the expected behavior

import elastic v5 verson

Please describe the actual behavior

import elastic v7 version and cannot find package such as below cannot find package "github.com/olivere/elastic/v7/config"

Any steps to reproduce the behavior?

Which version of Elastic are you using?

elastic.v2 (for Elasticsearch 1.x)

Please describe the expected behavior

Successful document put into the index.

Please describe the actual behavior

Error is returned:

elastic: Error 403 (Forbidden)

Any steps to reproduce the behavior?

Setup:

    creds := credentials.NewEnvCredentials()
    signer := v4.NewSigner(creds)
    awsClient, err := aws_signing_client.New(signer, nil, "es", "us-west-2")
    if err != nil {
        return nil, err
    }

    return elastic.NewClient(
        elastic.SetURL(...),
        elastic.SetScheme("https"),
        elastic.SetHttpClient(awsClient),
        elastic.SetSniff(false),
    )

Put:

    _, err = e.Client.Index().Index(indexName).Type(indexType).
        Id(doc.ID).
        BodyJson(doc).
        Do()

Not sure if this is elastic or aws_signing_client issue.

I am about to switch to go modules but it appears the branch release-branch.v6 has some import problems when you just try to use it using old-fashioned GOPATH...

When you compile you get:

/go/src/github.com/olivere/elastic/client.go:24:2: cannot find package "github.com/olivere/elastic/v6/config" in any of:
        /usr/local/go/src/github.com/olivere/elastic/v6/config (from $GOROOT)
        /go/src/github.com/olivere/elastic/v6/config (from $GOPATH)
/go/src/github.com/olivere/elastic/bulk.go:14:2: cannot find package "github.com/olivere/elastic/v6/uritemplates" in any of:
        /usr/local/go/src/github.com/olivere/elastic/v6/uritemplates (from $GOROOT)
        /go/src/github.com/olivere/elastic/v6/uritemplates (from $GOPATH)

I realize you recommend to use a dependency manager however prior to your latest update it still worked okay.

Version

elastic.v5 (for Elasticsearch 5.x)

How to reproduce:

package main

import (
	"context"
	"gopkg.in/olivere/elastic.v5"
	"log"
	"os"
	"time"
)

func main() {

	var err error

	client, err := elastic.NewClient(
		elastic.SetURL("https://httpbin.org/delay/3?"), // every request will take about 3 seconds
		elastic.SetHealthcheck(false),
		elastic.SetSniff(false),
		elastic.SetErrorLog(log.New(os.Stderr, "", log.LstdFlags)),
		elastic.SetInfoLog(log.New(os.Stdout, "", log.LstdFlags)),
	)
	if err != nil {
		log.Fatal(err)
	}

	ctx, _ := context.WithTimeout(context.Background(), 1*time.Second) // requests will time out after 1 second

	log.Println("Running request")

	_, err = client.Get().Index("whatever").Id("1").Do(ctx)

	if err != nil {
		log.Println("Error: " + err.Error())
	}

	log.Println("Running second request")

	_, err = client.Get().Index("whatever").Id("1").Do(ctx)

	if err != nil {
		log.Println("Error: " + err.Error())
	}

}

Actual

2017/03/17 08:02:33 Running request
2017/03/17 08:02:34 elastic: https://httpbin.org/delay/3? is dead
2017/03/17 08:02:34 Error: context deadline exceeded
2017/03/17 08:02:34 Running second request
2017/03/17 08:02:34 elastic: all 1 nodes marked as dead; resurrecting them to prevent deadlock
2017/03/17 08:02:34 Error: no Elasticsearch node available

Expected

Something like (I edited that "log" myself):

2017/03/17 08:02:33 Running request
2017/03/17 08:02:34 Error: context deadline exceeded
2017/03/17 08:02:34 Running second request
2017/03/17 08:02:37 GET https://httpbin.org/delay/3?/whatever/_all/1 [status:200, request:3.500s]

Hi!

Apologies if this is an already answered question, I was unable to find a satisfactory answer online. I am trying to find a way to write unit tests for one of my services however I feel this example could apply outside of unit tests to more general encapsulation of code.

I want to mock the client so I can simulate a specific type of request (in my case bulk requests) without going all the way to a test ElasticSearch instance. Ideally there would be an interface to allow me to generate a mock in my tests. For example what I want is an interface like so:

type IBulkClient interface {
    Bulk() IBulkService // return the BulkService interface
    ...
}

type IBulkService interface {
   Add(requests ...E.BulkableRequest) IBulkService
   Do() (IBulkResponse, error) // return the BulkResponse interface (not shown here)
   ...
}

This would allow me to mock BulkClient and BulkService to better test my code. The reason why I can't do this myself right now is that the real BulkService.Add() returns a *BulkService which screws up the interface as I want my IBulkClient to return another interface not a pointer to a struct.

Here is a go playground with the issue I am talking about and here is it working with an interface reference rather than a pointer to a struct.

My ultimate question is this: Is it possible for the API to provide interfaces for all its structs? This would allow for better unit testing and also allow the user to better encapsulate their code. If there is a reason why there shouldn't be interfaces how do you recommend writing unit tests that don't go all the way to ElasticSearch or mocking the response at an http level?

In Elasticsearch 7, hits.total is an object in the search response. This breaks searches in the current version of the library with an unmarshalling error: json: cannot unmarshal object into Go struct field SearchHits.total of type int64.

There is a request parameter (rest_total_hits_as_int=true) that can be added to get back the old behaviour, but I don't think this library currently has an easy way of adding this parameter to requests.

Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

elastic.v5 (for Elasticsearch 5.x)

Please describe the expected behavior

searchResult,err:= client().search().....Do(...)

searchResult.RawJson() // get full result raw json

Please describe the actual behavior

Did not find this method

Any steps to reproduce the behavior?

// Search with a term query termQuery := elastic.NewTermQuery("user", "olivere") searchResult, err := client.Search(). Index("twitter"). // search in index "twitter" Query(termQuery). // specify the query Sort("user", true). // sort by "user" field, ascending From(0).Size(10). // take documents 0-9 Pretty(true). // pretty print request and response JSON Do(context.Background()) // execute if err != nil { // Handle error panic(err) }

Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

[ ] elastic.v6 (for Elasticsearch 6.x)

Getting undefined error for NewV4SigningClient(cred,clusterLocation)

Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

[x] elastic.v7 (for Elasticsearch 7.x)

I'm having difficulty generating the desired histogram query with sub aggregations.

This is the query I want to generate.

{
  "size": 0,
  "aggs": {
    "range": {
      "histogram": {
        "field": "counter",
        "interval": 2
      },
      "aggs": {
        "nested": {
          "nested": {
            "path": "deposits"
          },
          "aggs": {
            "scripts": {
              "avg": {
                "script": {
                  "lang": "painless", 
                  "source": "return doc['deposits.depositA'].value + doc['deposits.depositB'].value"
                }
              }
            }
          }
        }
      }
    }
  }
}

You'll have to excuse me if this is a dumb question, since I'm new to both elasticsearch and this package. But this is as far as I've gotten:

nested := NewNestedAggregation().Path("deposits")

scriptq := elastic.NewScriptQuery(NewScript("return doc['deposits.depositA'].value + doc['deposits.depositB'].value"))
hist := NewHistogramAggregation().Field("counter").Interval(2).SubAggregation("avg", scriptq)

client.Search().Index(indexName).Aggregation() ... // this is where I get stuck

I'm not sure of the type of aggregation I can attach in Aggregation().

I've been reviewing the _test files of this package to get some idea on what to do but a lot of this is still going over my head.

Any help would be greatly appreciated.

Does olivere support the concept of REST API compatibility? Although it is a good practice not to make cross-communication between client and server which are of different versions, sometimes in practice, we encounter the need to use the same client to ping elasticsearch servers of different versions.

As mentioned in the page, in order to request REST API compatibility, the client needs to specify in the header about Accept and "Content-Types":

Accept: "application/vnd.elasticsearch+json;compatible-with=7"
Content-Type: "application/vnd.elasticsearch+json;compatible-with=7"

This is not a bug, but a question about current or potential future feature of the library. Does olivere already support this?

Which version of Elastic are you using?

[x] elastic.v7 (for Elasticsearch 7.x) [x] elastic.v6 (for Elasticsearch 6.x) [ ] elastic.v5 (for Elasticsearch 5.x) [ ] elastic.v3 (for Elasticsearch 2.x) [ ] elastic.v2 (for Elasticsearch 1.x)

Please describe the expected behavior

elastic v7 is able to make requests to both Elasticsearch 7 and 8 servers with the same endpoint, expecting the same behavior.

Please describe the actual behavior

NA, as this issue is a question rather than a bug.

Any steps to reproduce the behavior?

NA, as this issue is a question rather than a bug.

Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

[ x] elastic.v7 (for Elasticsearch 7.x)

Please describe the expected behavior

Once backoff is disabled, we expect all the data sent to the disabled (unavailable) cluster will be discarded, so we only process the response from the cluster in the after method, and store all the failed values into DLQ. Instead, the data sent at the time of the ES failure is kept and submitted once the connection to the ES cluster is restored.

Please describe the actual behavior

We have a constant flow of values, that are kept in different ES clusters. We have added the code that suppose to handle the case when of the ES clusters failed, which is sending that values to the dead letter queue (DLQ) and read them back to it when it is restored back. The issue we are having now - double records. Seems that even without the backoff policy defined (or something like stopBackoff{} one), BulkProcessor, once the ES cluster is restored sends all the values processed from the source into ES, at the same time as we are starting to process DLQ... This means that all the records ingested by the app while the ES cluster is off simply piled up on top of each other, instead of being discarded after the "after" method has been called. We've tried to overcome that issue by setting up StopBackoff like that:

processor, err = client.BulkProcessor().
			Workers(o.Workers).
			BulkActions(o.BatchSize).
			BulkSize(o.BatchBytes).
			FlushInterval(o.FlushInterval).
			RetryItemStatusCodes(o.RetryItemStatusCodes...).
                        Backoff(elastic.StopBackoff{}).
			Stats(o.WantStats).
			After(after). // call "after" after every commit
			Do(o.Ctx)

But that didn't help, since it seems to only generate errors one after another without doing much.

This behavior would've been fine if we were processing a low amount of messages, but when there are a lot of messages stored in the memory, eventually the app would be crashed and all the stored messages are lost.

Any steps to reproduce the behavior?

1. Set up the BulkProcessor with stopBackoff defined as backoff.
2. Disable ES cluster, while BulkProcessor app keeps running and ingest messages from the source (kafka)
3. Send some values to the app ingestion channel
4. Enable ES cluster
5. All the values sent to the BulkProcessor while the ES cluster was disabled are can be seen in the ES cluster.

Suggested solutions

I am seeing that we are only clearing records at one place, where we call s.Reset(), that part of the code is seems to be never reached if the cluster is off. Is there a way to clear out the records at the time after method has finished? Or add some setting that will allow doing so?

Which version of Elastic are you using?

[ ] elastic.v7 (for Elasticsearch 7.x)

Please describe the expected behavior

Client::dumpResponse should check for MaxResponseSize before dumping the request similar to what Client::newResponse does. Currently the library can cause subtle OOM situations even if the request is bounded by a MaxResponseSize limit unless the trace logger is also nil.

Consumers should be able to set a trace logger without risking OOMs. At the very least it should be made clear in the documentation that MaxResponseSize must be set and the trace logger must be nil to guard against OOM exceptions.

Please describe the actual behavior

Library can cause OOM exceptions if trace logger is set, regardless of the MaxResponseSize limits.

Any steps to reproduce the behavior?

Instantiate a client and set both the MaxResponseSize and the trace logger to be non-zero valued. You will see that the process will use a lot of memory even if you set the MaxResponseSize to be a low value.

[x] elastic.v6 (for Elasticsearch 6.x)

Please describe the expected behavior

I need to search an email

Please describe the actual behavior

without the token uax_url_email sent to E/S, my email is not found

How to add a tokenizer uax_url_email ?