convert sql to elasticsearch DSL in golang(go)

 _____ _         _     ____ _____ ___  ____  ____   ___   _
| ____| |       / \   / ___|_   _|_ _|/ ___|/ ___| / _ \ | |
|  _| | |      / _ \  \___ \ | |  | || |    \___ \| | | || |
| |___| |___  / ___ \  ___) || |  | || |___  ___) | |_| || |___
|_____|_____|/_/   \_\|____/ |_| |___|\____||____/ \__\_\|_____|

Overview

Build Status Go Documentation Coverage Status Go Report Card

This tool converts sql to elasticsearch dsl

Currently support:

  • sql and expression
  • sql or expression
  • equal(=) support
  • not equal(!=) support
  • gt(>) support
  • gte(>=) support
  • lt(<) support
  • lte(<=) support
  • sql in (eg. id in (1,2,3) ) expression
  • sql not in (eg. id not in (1,2,3) ) expression
  • paren bool support (eg. where (a=1 or b=1) and (c=1 or d=1))
  • sql like expression (currently use match phrase, perhaps will change to wildcard in the future)
  • sql order by support
  • sql limit support
  • sql not like expression
  • field missing check
  • support aggregation like count(*), count(field), min(field), max(field), avg(field)
  • support aggregation like stats(field), extended_stats(field), percentiles(field) which are not standard sql function
  • null check expression(is null/is not null)
  • join expression
  • having support

Usage

go get -u github.com/cch123/elasticsql

Demo :

package main

import (
    "fmt"

    "github.com/cch123/elasticsql"
)

var sql = `
select * from aaa
where a=1 and x = '三个男人'
and create_time between '2015-01-01T00:00:00+0800' and '2016-01-01T00:00:00+0800'
and process_id > 1 order by id desc limit 100,10
`

func main() {
    dsl, esType, _ := elasticsql.Convert(sql)
    fmt.Println(dsl)
    fmt.Println(esType)
}

will produce :

{
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "a": {
                            "query": "1",
                            "type": "phrase"
                        }
                    }
                },
                {
                    "match": {
                        "x": {
                            "query": "三个男人",
                            "type": "phrase"
                        }
                    }
                },
                {
                    "range": {
                        "create_time": {
                            "from": "2015-01-01T00:00:00+0800",
                            "to": "2016-01-01T00:00:00+0800"
                        }
                    }
                },
                {
                    "range": {
                        "process_id": {
                            "gt": "1"
                        }
                    }
                }
            ]
        }
    },
    "from": 100,
    "size": 10,
    "sort": [
        {
            "id": "desc"
        }
    ]
}

aaa

If your sql contains some keywords, eg. order, timestamp, don't forget to escape these fields as follows:

select * from `order` where `timestamp` = 1 and `desc`.id > 0

Warning

To use this tool, you need to understand the term query and match phrase query of elasticsearch.

Setting a field to analyzed or not analyzed will get different results.

Details

For more details of convertion, please refer to the wiki

Other info

When writing this tool, I tried to avoid the deprecated dsl filters and aggregations, so it is compatible with most versions of the elasticsearch

If you have any advices or ideas, welcome to submit an issue or Pull Request!

License

MIT

Owner
Xargin
If you don't keep moving, you'll quickly fall behind.
Xargin
Comments
  • 简单查询语句报错

    简单查询语句报错

    select * from aaa where remote_addr="111.206.36.1" limit 10; 转换为: {"query" : {"bool" : {"must" : [{"match" : {"remote_addr" : {"query" : "111.206.36.1", "type" : "phrase"}}}]}},"from" : 0,"size" : 10}

    用该dsl语句查询的时候: { "error": { "root_cause": [ { "type": "parsing_exception", "reason": "[match] query does not support [type]", "line": 1, "col": 84 } ], "type": "parsing_exception", "reason": "[match] query does not support [type]", "line": 1, "col": 84 }, "status": 400 }

    必须要删除 "type" : "phrase" 才行

  • 能否支持escape?

    能否支持escape?

    原始sql

    SELECT
        `familyCode`
    FROM
        `myTable`
    WHERE
        (`admin` LIKE '%v!_xxx%' ESCAPE '!')
    ORDER BY
        `updateTime` DESC
    LIMIT
        10
    

    其中要对admin字段双侧LIKE字符串v_xxx。 输出的结果:

    {
        "query": {
            "bool": {
                "must": [
                    {
                        "match_phrase": {
                            "admin": {
                                "query": "v!_xxx"
                            }
                        }
                    }
                ]
            }
        },
        "from": 0,
        "size": 10,
        "sort": [
            {
                "updateTime": "desc"
            }
        ]
    }
    

    其中query.bool.must[0].match_phrase.admin.query字段的值中,! 没有去掉。

  • Large Query

    Large Query

    Hello i need to know when we make it large query have alot of join inside SQL possible to make by your Go solution to Elastic Query match ?

    Thanks, George

  • Is the example correct?

    Is the example correct?

    First of all thanks for the great work!

    The functionality check-list mentions that like operator is implemented for phrase search.

    However, in the example, although the strict equality = is used, the generated elasticsearch query is a phrase match. In that case, as far as I understood, shouldn't it generate a term search instead?

    If the example is correct, what is the way to generate a term search query?

  • 建议在Convert的时候,可以加入json pretty输出

    建议在Convert的时候,可以加入json pretty输出

    	if Pretty {
    		var prettyJSON bytes.Buffer
    		err = json.Indent(&prettyJSON, []byte(dsl), "", "  ")
    		if err != nil {
    			dsl = ""
    		} else {
    			dsl = string(prettyJSON.Bytes())
    		}
    	}
    
  • is it possible to make group by every 5 minutes or somethings?

    is it possible to make group by every 5 minutes or somethings?

    elaticsearch sql have below dsl, { "aggs" : { "articles_over_time" : { "date_histogram" : { "field" : "date", "interval" : "1.5h" } } } }

    is it possible?

  • date_histogram转换DSL有问题?

    date_histogram转换DSL有问题?

    使用例程转换下列语句为DSL的时候,来源
    select * from table_name group by date_histogram(field='create_time', interval='1h') 报错 syntax error at position 81

    interval要改成value么?

  • convert error : nend convert select sql fields to json fields like `

    convert error : nend convert select sql fields to json fields like `"_source":["title", "price"]`

    original SQL :

    select title, price from products where title = '%A%'
    

    expect like that :

    {
        "_source": [
            "title",
            "price",
        ],
        "query": {
            "bool": {
                "must": [
                    {
                        "match_phrase": {
                            "title": {
                                "query": "%A%"
                            }
                        }
                    }
                ]
            }
        },
        "from": 0,
        "size": 1
    } 
    

    actually, elasticsql.Convert(sql) didn't generate "_source":["title", "price"]

  • Multiple Group by errors

    Multiple Group by errors

    sql like : select count(id) from A where A.id > 5 group by A.a, A.b; because of innerMap["terms"] = msi{ "field": colName.Name.String(), "size": 0, } will gen the dsl like: "terms":{"field":"field2","size":0} and then make a error: "reason":"[size] must be greater than 0 using es7.X

A river for elasticsearch to automatically index mysql content using the replication feed.

Mysql River Plugin for ElasticSearch The Mysql River plugin allows to hook into Mysql replication feed using the excellent python-mysql-replication an

Jun 1, 2022
Zinc Search engine. A lightweight alternative to elasticsearch that requires minimal resources, written in Go.
Zinc Search engine. A lightweight alternative to elasticsearch that requires minimal resources, written in Go.

Zinc Zinc is a search engine that does full text indexing. It is a lightweight alternative to elasticsearch and runs in less than 100 MB of RAM. It us

Jan 8, 2023
This package can parse date match expression, which used by ElasticSearch

datemath-parser this package is pure go package, this package can parse date match expression, which used by ElasticSearch. Date Math Definition you c

Jan 8, 2022
write APIs using direct SQL queries with no hassle, let's rethink about SQL

SQLer SQL-er is a tiny portable server enables you to write APIs using SQL query to be executed when anyone hits it, also it enables you to define val

Jan 7, 2023
Parses a file and associate SQL queries to a map. Useful for separating SQL from code logic

goyesql This package is based on nleof/goyesql but is not compatible with it any more. This package introduces support for arbitrary tag types and cha

Oct 20, 2021
Go-sql-reader - Go utility to read the externalised sql with predefined tags

go-sql-reader go utility to read the externalised sql with predefined tags Usage

Jan 25, 2022
Convert data exports from various services to a single SQLite database
Convert data exports from various services to a single SQLite database

Bionic Bionic is a tool to convert data exports from web apps to a single SQLite database. Bionic currently supports data exports from Google, Apple H

Dec 9, 2022
a golang library for sql builder

Gendry gendry is a Go library that helps you operate database. Based on go-sql-driver/mysql, it provides a series of simple but useful tools to prepar

Dec 26, 2022
Fluent SQL generation for golang

Squirrel is "complete". Bug fixes will still be merged (slowly). Bug reports are welcome, but I will not necessarily respond to them. If another fork

Dec 29, 2022
GoTSQL : A Better Way to Organize SQL codebase using Templates in Golang

GoTSQL - A Better Way to Organize SQL codebase using Templates in Golang Installation through Go Get command $ go get github.com/migopsrepos/gotsql In

Aug 17, 2022
SQL transaction wrapper on golang

TxWrapper TxWrapper is a sql transaction wrapper. It helps to exclude writing code for rollback and commit commands. Usage import ( "context"

Mar 14, 2022
Simple SQL escape and format for golang

sqlstring Simple SQL escape and format Escaping sql values //Format sql := sqlstring.Format("select * from users where name=? and age=? limit ?,?", "t

Sep 4, 2022
BigQuery database/sql golang driver

BigQuery SQL Driver This library is compatible with Go 1.17+ Please refer to CHA

Dec 7, 2022
Spansqlx - Spanner sql pkgs with golang

spansqlx spanner sql pkgs install go get github.com/reiot777/spansqlx usage Bel

Jan 15, 2022
Golang REST Layer SQL Storage Handler

This REST Layer resource storage backend stores data in a SQL Database using database/sql.

Feb 15, 2022
Go package for sharding databases ( Supports every ORM or raw SQL )
Go package for sharding databases ( Supports every ORM or raw SQL )

Octillery Octillery is a Go package for sharding databases. It can use with every OR Mapping library ( xorm , gorp , gorm , dbr ...) implementing data

Dec 16, 2022
Prep finds all SQL statements in a Go package and instruments db connection with prepared statements

Prep Prep finds all SQL statements in a Go package and instruments db connection with prepared statements. It allows you to benefit from the prepared

Dec 10, 2022
pggen - generate type safe Go methods from Postgres SQL queries

pggen - generate type safe Go methods from Postgres SQL queries pggen is a tool that generates Go code to provide a typesafe wrapper around Postgres q

Jan 3, 2023
🐳 A most popular sql audit platform for mysql
🐳 A most popular sql audit platform for mysql

?? A most popular sql audit platform for mysql

Jan 6, 2023