go-explosion
Distributed indexing and searching in Go/Golang.
This library does not work yet, it is in development.
Introduction
go-explosion allows you to distribute indexing and searching using HashiCorp Consul, HashiCorp Nomad, MinIO and Bluge
The HTTP API of go-explosion is compatible with Elasticsearch (Elasticsearch Specification). Because of this compatibility we make use of the official go-elasticsearch client and optionally esquery for building queries.
Warning: Encryption is currently not supported since Consul does not support looking up individual nodes: https://github.com/hashicorp/consul/issues/11904
Features
- Modern Go indexing library
- Compatible and integrates with Elasticsearch APIs
- Create an index with shards and replicas.
- Supports text and keyword mapping types
- HashiCorp Consul
- Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
- Distributed key-value store.
- Encrypted communications using TLS.
- HashiCorp Nomad
- Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
- MinIO
- High Performance Object Storage.
- The API is compatible with Amazon S3 cloud storage service.
Installation
Download HashiCorp Consul.
# Start Consul
$ ./consul agent -server -bootstrap -data-dir=data -bind=127.0.0.1 --config-file ~/path/to/go-explosion/consul.hcl -ui
Download MinIO.
# Start MinIO
$ MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=yourrootpassword ./minio server data/ --console-address ":9001"
# The console can be accessed via the browser at http://127.0.0.1:9001
Download go-explosion.
# Clone or download go-explosion
$ git clone https://github.com/mooijtech/go-explosion
# Change directory
$ cd go-explosion
Usage
Start the server
Note: The server should be started via HashiCorp Nomad.
package main
import explosion "github.com/mooijtech/go-explosion/pkg"
func main() {
server := explosion.NewServer()
if err := server.Start("127.0.0.1", 1337); err != nil {
explosion.Logger.Error("Failed to start server: %s", err)
}
}
Create an index
package main
import (
"bytes"
"encoding/json"
explosion "github.com/mooijtech/go-explosion/pkg"
"io/ioutil"
)
func main() {
client, err := explosion.NewDefaultClient()
if err != nil {
explosion.Logger.Error("Failed to create client: %s", err)
return
}
var requestBody bytes.Buffer
err = json.NewEncoder(&requestBody).Encode(map[string]interface{} {
"settings": map[string]interface{} {
"number_of_shards": 3,
"number_of_replicas": 2,
},
// References:
// https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html
// https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html
"mappings": map[string]interface{} {
"properties": map[string]interface{} {
"name": map[string]interface{} {
"type": "text",
},
"email": map[string]interface{} {
"type": "keyword",
},
"phone": map[string]interface{} {
"type": "text",
},
},
},
})
if err != nil {
explosion.Logger.Error("Failed to encode request body: %s", err)
return
}
response, err := client.Elasticsearch.Indices.Create("explosion", client.ElasticSearch.Indices.Create.WithBody(&requestBody))
if err != nil {
explosion.Logger.Error("Failed to get response: %s", err)
return
}
body, err := ioutil.ReadAll(response.Body)
if err != nil {
explosion.Logger.Error("Failed to read response body: %s", err)
return
}
explosion.Logger.Info("Successfully created index: %s", body)
}
Index a document
package main
import (
"bytes"
"encoding/json"
explosion "github.com/mooijtech/go-explosion/pkg"
"io/ioutil"
)
func main() {
client, err := explosion.NewDefaultClient()
if err != nil {
explosion.Logger.Error("Failed to create client: %s", err)
return
}
var requestBody bytes.Buffer
err = json.NewEncoder(&requestBody).Encode(map[string]interface{} {
"name": "Marten Mooij",
"email": "[email protected]",
"phone": "+31 6 30 53 47 67",
})
if err != nil {
explosion.Logger.Error("Failed to encode request body: %s", err)
return
}
response, err := client.Elasticsearch.Index("explosion", &requestBody)
if err != nil {
explosion.Logger.Error("Failed to get response: %s", err)
return
}
body, err := ioutil.ReadAll(response.Body)
if err != nil {
explosion.Logger.Error("Failed to read response body: %s", err)
return
}
explosion.Logger.Info("Successfully created index: %s", body)
}
References
- Elasticsearch Specification
- Elasticsearch from the Bottom Up, Part 1
- Elasticsearch from the Top Down
- Create index API
- Index API
Libraries
Contact
Name: Marten Mooij
Email: [email protected]
Phone: +31 6 30 53 47 67
