PostgreSQL style Parser splitted from CockroachDB

What's this

PostgreSQL style Parser splitted from CockroachDB

See: Complex SQL format example

I tried to import github.com/cockroachdb/cockroach/pkg/sql/parser, but the dependencies is too complex to make it work.

To make things easy, I did these things:

  1. Copy all the pkg/sql/parser, pkg/sql/lex and simplify the dependencies
  2. Simplify the Makefile to just generate the goyacc stuff
  3. Add the goyacc generated files in parser and lex to make go get work easily, see the .gitignore files
  4. Trim the etcd dependency, see the go.mod
  5. Rename all test file except some pkg/sql/parser tests
  6. Add all necessary imports to vendor
  7. Remove the panic of meeting unregistried functions, see the WrapFunction
  8. Other nasty things make the parser just work that I forgot :p

Features

  • Pure golang implementation
  • Almost full support of PostgreSQL (cockroachdb style PostgreSQL)

SQL Standard Compliance

The code is derived from CockroachDB v20.1.11 which supports most of the major features of SQL:2011. See:

🚧 🚧 🚧 still under construction 🚧 🚧 🚧

Progress

  • 2020-02-16 github.com/auxten/postgresql-parser/pkg/sql/parser Unit tests works now!

Todo

  • Fix more unit tests
  • Make built-in function parsing work
Comments
  • Update dependencies to latest version

    Update dependencies to latest version

    I updated all the dependencies to the latest versions and removed the replaced dependency.

    @auxten Please let me know if this is not OK and if you based the project on certain dependencies. Then we can pin those dependencies. Also, I am thinking about various ways to contribute to this project. Please let me know of the next steps or create tickets, so we can talk about various aspects of it.

  • cleaning up Makefile and various bug fixes inside the build processes

    cleaning up Makefile and various bug fixes inside the build processes

    This PR removes most of the unused/dead Makefile code relevant to the CockroachDB project. This PR also fixes some small issues inside the build processes such as the import paths defined inside sql.y and the non-alphanumeric build constraint.

    All generated files could be generated using make generate. Make generate does not yet ensure that all the necessary tools are installed. This will be included in a future PR.

    NOTE: the vendor directory inside the project is currently outdated. I would like to suggest removing the vendor directory since all import issues have been resolved inside this PR.

  • run make all get error   no /pkg/cmd/prereqs

    run make all get error no /pkg/cmd/prereqs

    (echo "// Code generated by goyacc. DO NOT EDIT.";
    echo "// GENERATED FILE DO NOT EDIT";
    cat pkg/sql/parser/gen/sql.go.tmp |
    sed -E 's/^const ([A-Z][_A-Z0-9]) =.$/const \1 = lex.\1/g') > pkg/sql/parser/sql.go.tmp || rm pkg/sql/parser/sql.go.tmp mv -f pkg/sql/parser/sql.go.tmp pkg/sql/parser/sql.go goimports -w pkg/sql/parser/sql.go (echo "// Code generated by make. DO NOT EDIT.";
    echo "// GENERATED FILE DO NOT EDIT";
    echo;
    echo "package lex";
    echo;
    grep '^const [A-Z][_A-Z0-9]* ' pkg/sql/parser/gen/sql.go.tmp) > pkg/sql/lex/tokens.go.tmp || rm pkg/sql/lex/tokens.go.tmp mv -f pkg/sql/lex/tokens.go.tmp pkg/sql/lex/tokens.go go install -v ./pkg/cmd/prereqs can't load package: package ./pkg/cmd/prereqs: cannot find package "." in:

  • Dependency security warning

    Dependency security warning

    I am getting a high severity security warning, can we update this dependency?

    Improper Input Validation in GoGo Protobuf

    Package Affected versions Patched version
    github.com/gogo/protobuf(Go) < 1.3.2 1.3.2

    An issue was discovered in GoGo Protobuf before 1.3.2. plugin/unmarshal/unmarshal.go lacks certain index validation, aka the "skippy peanut butter" issue.

  • 无法解析带有类型转换符

    无法解析带有类型转换符 "::" 的语句

    解析此种类型的sql失败:

    SELECT COUNT ( * ) FROM pg_class C LEFT JOIN pg_namespace n ON n.oid = C.relnamespace WHERE C.relkind = ANY ( '{r,v,m}' :: CHAR [] ) UNION SELECT COUNT ( * ) FROM pg_attribute A JOIN pg_class C ON A.attrelid = C.oid JOIN pg_namespace n ON C.relnamespace = n.oid JOIN pg_type tp ON tp.typelem = A.atttypid WHERE A.attnum > 0 UNION SELECT COUNT ( * ) FROM information_schema.routines

    出错的位置在 "::" 操作符

  • hlc_clock_device_linux.go

    hlc_clock_device_linux.go

    github.com/auxten/postgresql-parser/pkg/util/hlc ../../auxten/postgresql-parser/pkg/util/hlc/hlc_clock_device_linux.go:51:11: cannot use ctx (type context.Context) as type string in argument to logrus.Infof

    Using arch. Commented out that single log write on line 51 and 'log' import and the example worked.

    Thanks!

  • build: upgrade dep version

    build: upgrade dep version

    Desc

    Upgrade github.com/gogo/protobuf version close #17

    Note

    I just upgrade package version and i don't know how to test it. I only tried example to be executable.

  • Switch

    Switch "STRING" for "TEXT" string type by default.

    CockroachDB uses "STRING" (but accepts "TEXT" for compatibility) and PostgreSQL uses "TEXT" always.

    Fixes https://github.com/auxten/postgresql-parser/issues/15

  • Removed the unused storage engine protobuffers

    Removed the unused storage engine protobuffers

    I am not sure if you are planning to use this code for future features. I would like to suggest to remove the storage package if there are no plans made for this package.

  • Move examples into separate directories

    Move examples into separate directories

    go vet currently throws the error that main is redeclared inside the example directory. This PR fixes this issue by moving the examples into separate directories.

  • Data type STRING not recognized

    Data type STRING not recognized

    Hi,

    I'm experimenting with postgresql-parser to intercept and rewrite some queries to a postgresql database.

    One problem I have encountered is that the formatter will output "STRING" by default for string data type. PostgreSQL does not support this, instead the data type name is "TEXT". CockroachDB supports "TEXT" as an alias for "STRING" [1]

    In order to make query-rewriting work, I've patched postgresql-parser to output "TEXT" data type by default instead of "STRING". I think this could be a reasonable default since CockroachDB supports both.

    [1] https://www.cockroachlabs.com/docs/stable/string.html

  • Data type STRING not recognized

    Data type STRING not recognized

    Hi,

    I'm experimenting with postgresql-parser to intercept and rewrite some queries to a postgresql database.

    One problem I have encountered is that the formatter will output "STRING" by default for string data type. PostgreSQL does not support this, instead the data type name is "TEXT". CockroachDB supports "TEXT" as an alias for "STRING" [1]

    In order to make query-rewriting work, I've patched postgresql-parser to output "TEXT" data type by default instead of "STRING". I think this could be a reasonable default since CockroachDB supports both.

    [1] https://www.cockroachlabs.com/docs/stable/string.html

pg_timetable: Advanced scheduling for PostgreSQL
pg_timetable: Advanced scheduling for PostgreSQL

pg_timetable: Advanced scheduling for PostgreSQL pg_timetable is an advanced job scheduler for PostgreSQL, offering many advantages over traditional s

Dec 29, 2022
Cross-platform client for PostgreSQL databases

pgweb Web-based PostgreSQL database browser written in Go. Overview Pgweb is a web-based database browser for PostgreSQL, written in Go and works on O

Dec 30, 2022
pREST (PostgreSQL REST), simplify and accelerate development, ⚡ instant, realtime, high-performance on any Postgres application, existing or new

pREST pREST (PostgreSQL REST), simplify and accelerate development, instant, realtime, high-performance on any Postgres application, existing or new P

Jan 9, 2023
Enhanced PostgreSQL logical replication

pgcat - Enhanced postgresql logical replication Why pgcat? Architecture Build from source Install Run Conflict handling Table mapping Replication iden

Dec 21, 2022
Interactive client for PostgreSQL and MySQL
Interactive client for PostgreSQL and MySQL

dblab Interactive client for PostgreSQL and MySQL. Overview dblab is a fast and lightweight interactive terminal based UI application for PostgreSQL a

Jan 8, 2023
WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

Jan 1, 2023
Worker failover support for PostgreSQL Citus extension using pg_auto_failover.

citus-failover Worker failover support for citus community version using pg_auto_failover. What is this? This is a simple service to monitor changes i

Dec 7, 2022
Interactive terminal user interface and CLI for database connections. MySQL, PostgreSQL. More to come.
Interactive terminal user interface and CLI for database connections. MySQL, PostgreSQL. More to come.

?? dbui dbui is the terminal user interface and CLI for database connections. It provides features like, Connect to multiple data sources and instance

Jan 5, 2023
Modify orca-zhang/borm in order to use in PostgreSQL

borm ??️ 针对 orca-zhang/borm 进行了修改,暂时只能兼容PostgreSQL 原因 在b站时候用过borm,用起来感觉非常简洁 自己学校里用PostgreSQL比较多 可变条件真的非常好用 问题 首先需要注意的是,这是写给PG的 PG 根本不存在某些 MySQL 独有的函数

Aug 24, 2022
Cross-platform client for PostgreSQL databases

pgweb Web-based PostgreSQL database browser written in Go. Overview Pgweb is a web-based database browser for PostgreSQL, written in Go and works on O

Dec 30, 2022
PolarDB Cluster Manager is the cluster management component of PolarDB for PostgreSQL, responsible for topology management, high availability, configuration management, and plugin extensions.

What is PolarDB Cluster Manager PolarDB Cluster Manager is the cluster management component of PolarDB for PostgreSQL, responsible for topology manage

Nov 9, 2022
A simple Golang-based application that queries a PostgreSQL database

Qwik-E-Mart Demo App A simple Golang-based application that queries a PostgreSQL database named qwikemart to read and return customer data stored in t

Nov 6, 2021
CloudQuery extracts, transforms, and loads your cloud assets into normalized PostgreSQL tables.
CloudQuery extracts, transforms, and loads your cloud assets into normalized PostgreSQL tables.

The open-source cloud asset inventory backed by SQL. CloudQuery extracts, transforms, and loads your cloud assets into normalized PostgreSQL tables. C

Dec 31, 2022
Implemented PostgreSQL with Golang
Implemented PostgreSQL with Golang

Customer Information Web Api Implemented PostgreSQL with Golang docker run --name postgresql-container -p 5432:5432 -e POSTGRES_PASSWORD=Password! -d

Nov 15, 2021
Typescript type declaration to PostgreSQL CREATE TABLE converter

ts2psql NOTE: This is WIP. Details in this readme are ideal state. Current usage: go build && ./ts2psql (or go build && ts2psql if on Windows OS). A s

Jan 13, 2022
An experimental toolkit for injecting alternate authentication strategies into a PostgreSQL-compatible wire format.

PG Auth Proxy This is an experimental toolkit for injecting alternate authentication strategies into a PostgreSQL-compatible wire format. This is a pr

Jan 20, 2022
Thin clones of PostgreSQL to build powerful development, test, QA, staging environments
 Thin clones of PostgreSQL to build powerful development, test, QA, staging environments

Database Lab Engine (DLE) ⚡ Blazing-fast cloning of PostgreSQL databases ?? Thin clones of PostgreSQL to build powerful development, test, QA, staging

Jan 3, 2023
Dugopg - PostgreSQL tool For Golang

⚡️ DuGoPG Installation go get -u github.com/durudex/dugopg Example import (

May 9, 2022
📊 parallel black box PostgreSQL unit tests run against a real database.
📊   parallel black box PostgreSQL unit tests run against a real database.

?? psql-docker-tests-example Parallel black box PostgreSQL unit tests run against a real database. Consider reading the Medium Story first. This packa

Sep 15, 2022