Additions to Go's database/sql for super fast performance and convenience. (fork of gocraft/dbr)

dbr (fork of gocraft/dbr) provides additions to Go's database/sql for super fast performance and convenience.

Build Status Go Report Card Coverage Status

Getting Started

// create a connection (e.g. "postgres", "mysql", or "sqlite3")
conn, _ := dbr.Open("postgres", "...")

// create a session for each business unit of execution (e.g. a web request or goworkers job)
sess := conn.NewSession(nil)

// get a record
var suggestion Suggestion
sess.Select("id", "title").From("suggestions").Where("id = ?", 1).Load(&suggestion)

// JSON-ready, with dbr.Null* types serialized like you want
json.Marshal(&suggestion)

Feature highlights

Use a Sweet Query Builder or use Plain SQL

mailru/dbr supports both.

Sweet Query Builder:

stmt := dbr.Select("title", "body").
	From("suggestions").
	OrderBy("id").
	Limit(10)

Plain SQL:

builder := dbr.SelectBySql("SELECT `title`, `body` FROM `suggestions` ORDER BY `id` ASC LIMIT 10")

Amazing instrumentation with session

All queries in mailru/dbr are made in the context of a session. This is because when instrumenting your app, it's important to understand which business action the query took place in.

Writing instrumented code is a first-class concern for mailru/dbr. We instrument each query to emit to a EventReceiver interface.

Faster performance than using database/sql directly

Every time you call database/sql's db.Query("SELECT ...") method, under the hood, the mysql driver will create a prepared statement, execute it, and then throw it away. This has a big performance cost.

mailru/dbr doesn't use prepared statements. We ported mysql's query escape functionality directly into our package, which means we interpolate all of those question marks with their arguments before they get to MySQL. The result of this is that it's way faster, and just as secure.

Check out these benchmarks.

IN queries that aren't horrible

Traditionally, database/sql uses prepared statements, which means each argument in an IN clause needs its own question mark. mailru/dbr, on the other hand, handles interpolation itself so that you can easily use a single question mark paired with a dynamically sized slice.

ids := []int64{1, 2, 3, 4, 5}
builder.Where("id IN ?", ids) // `id` IN ?

map object can be used for IN queries as well. Note: interpolation map is slower than slice and it is preferable to use slice when it is possible.

ids := map[int64]string{1: "one", 2: "two"}
builder.Where("id IN ?", ids)  // `id` IN ?

JSON Friendly

Every try to JSON-encode a sql.NullString? You get:

{
	"str1": {
		"Valid": true,
		"String": "Hi!"
	},
	"str2": {
		"Valid": false,
		"String": ""
  }
}

Not quite what you want. mailru/dbr has dbr.NullString (and the rest of the Null* types) that encode correctly, giving you:

{
	"str1": "Hi!",
	"str2": null
}

Inserting multiple records

sess.InsertInto("suggestions").Columns("title", "body").
  Record(suggestion1).
  Record(suggestion2)

Updating records on conflict

stmt := sess.InsertInto("suggestions").Columns("title", "body").Record(suggestion1)
stmt.OnConflict("suggestions_pkey").Action("body", dbr.Proposed("body"))

Updating records

sess.Update("suggestions").
	Set("title", "Gopher").
	Set("body", "I love go.").
	Where("id = ?", 1)

Transactions

tx, err := sess.Begin()
if err != nil {
  return err
}
defer tx.RollbackUnlessCommitted()

// do stuff...

return tx.Commit()

Load database values to variables

Querying is the heart of mailru/dbr.

  • Load(&any): load everything!
  • LoadStruct(&oneStruct): load struct
  • LoadStructs(&manyStructs): load a slice of structs
  • LoadValue(&oneValue): load basic type
  • LoadValues(&manyValues): load a slice of basic types
// columns are mapped by tag then by field
type Suggestion struct {
	ID int64  // id, will be autoloaded by last insert id
	Title string // title
	Url string `db:"-"` // ignored
	secret string // ignored
	Body dbr.NullString `db:"content"` // content
	User User
}

// By default dbr converts CamelCase property names to snake_case column_names
// You can override this with struct tags, just like with JSON tags
// This is especially helpful while migrating from legacy systems
type Suggestion struct {
	Id        int64
	Title     dbr.NullString `db:"subject"` // subjects are called titles now
	CreatedAt dbr.NullTime
}

var suggestions []Suggestion
sess.Select("*").From("suggestions").Load(&suggestions)

Join multiple tables

dbr supports many join types:

sess.Select("*").From("suggestions").
  Join("subdomains", "suggestions.subdomain_id = subdomains.id")

sess.Select("*").From("suggestions").
  LeftJoin("subdomains", "suggestions.subdomain_id = subdomains.id")

sess.Select("*").From("suggestions").
  RightJoin("subdomains", "suggestions.subdomain_id = subdomains.id")

sess.Select("*").From("suggestions").
  FullJoin("subdomains", "suggestions.subdomain_id = subdomains.id")

You can join on multiple tables:

sess.Select("*").From("suggestions").
  Join("subdomains", "suggestions.subdomain_id = subdomains.id").
  Join("accounts", "subdomains.accounts_id = accounts.id")

Quoting/escaping identifiers (e.g. table and column names)

dbr.I("suggestions.id") // `suggestions`.`id`

Subquery

sess.Select("count(id)").From(
  dbr.Select("*").From("suggestions").As("count"),
)

Union

dbr.Union(
  dbr.Select("*"),
  dbr.Select("*"),
)

dbr.UnionAll(
  dbr.Select("*"),
  dbr.Select("*"),
)

Union can be used in subquery.

Alias/AS

  • SelectStmt
dbr.Select("*").From("suggestions").As("count")
  • Identity
dbr.I("suggestions").As("s")
  • Union
dbr.Union(
  dbr.Select("*"),
  dbr.Select("*"),
).As("u1")

dbr.UnionAll(
  dbr.Select("*"),
  dbr.Select("*"),
).As("u2")

Building arbitrary condition

One common reason to use this is to prevent string concatenation in a loop.

  • And
  • Or
  • Eq
  • Neq
  • Gt
  • Gte
  • Lt
  • Lte
dbr.And(
  dbr.Or(
    dbr.Gt("created_at", "2015-09-10"),
    dbr.Lte("created_at", "2015-09-11"),
  ),
  dbr.Eq("title", "hello world"),
)

Built with extensibility

The core of dbr is interpolation, which can expand ? with arbitrary SQL. If you need a feature that is not currently supported, you can build it on your own (or use dbr.Expr).

To do that, the value that you wish to be expaned with ? needs to implement dbr.Builder.

type Builder interface {
	Build(Dialect, Buffer) error
}

Driver support

  • MySQL
  • PostgreSQL
  • SQLite3
  • ClickHouse

These packages were developed by the engineering team at UserVoice and currently power much of its infrastructure and tech stack.

Thanks & Authors

Inspiration from these excellent libraries:

  • sqlx - various useful tools and utils for interacting with database/sql.
  • Squirrel - simple fluent query builder.

Authors:

Contributors:

Owner
Free and open source software developed at Mail.Ru
Free and open source software developed at Mail.Ru
Comments
  • can't use with go modules

    can't use with go modules

    When I'll try to get latest stable version of mailru/dbr:

    module some/project
    
    require (
    	...
    	github.com/mailru/dbr v0.0.0-20180825184916-bee39f330c3b
    	...
    )
    

    Problem is in release-tag naming:

    vX.Y with vX.Y.Z

  • Versioning

    Versioning

    Could you please create a new GitHub release or just version tag, because dependency managers like the official dep (https://github.com/golang/dep) use those versions by default.

  • Port 9000 is for clickhouse-client program

    Port 9000 is for clickhouse-client program

    Hello guys, I don't understand what I'm doing wrong, this IS a client program after all. With sqlx I'm using "tcp://host:9000" and it works correctly, we're evaluating using dbr instead but it complains: unsupported protocol scheme "tcp"

    I've tried a bunch of things already, setting only host/default it complains until I call Load: panic: runtime error: index out of range [0] with length 0 Actually, it doesn't matter what I put as host in Open(), I even tried with foo.var as host and it ran all the way to the call of query.Load() xD

    I tried with and without user, different ports, https scheme, I don't know what else to try, I haven't found any different examples here or otherwise.

    Any suggestions?

    package main
    
    import (
    	"log"
    
    	"github.com/mailru/dbr"
    	_ "github.com/mailru/go-clickhouse"
    )
    
    func main() {
    	connect, err := dbr.Open("clickhouse", "http://default@host:9000/default", nil)
    	if err != nil {
    		log.Fatalln(err)
    	}
    	sess := connect.NewSession(nil)
    	query := sess.Select("start").From("sometable").Where(dbr.Eq("id", 3))
    
    	var result []struct {
    		Start string `db:"start"`
    	}
    	count, err := query.Load(&result)
    	if err != nil {
    		log.Fatalln(err)
    	}
    
    	log.Printf("count %d  select %v\n", count, result)
    }
    
  • allow passing sql.TxOptions when using Session.Begin

    allow passing sql.TxOptions when using Session.Begin

    Sometimes when using transactions I want to enforce behavior by setting read-only or isolation level. The current Session.Begin doesn't take any options and when it eventually calls BeginTx to get the underlying sql.Tx it just uses nil for the options.

    This change is backward compatible and allows passing sql.TxOptions to BeginWithOpts while Begin retains it's original functionality.

    Usage:

    tx, err := sess.BeginWithOpts(&sql.TxOptions{
        Isolation: sql.LevelRepeatableRead,
    })
    defer tx.RollbackUnlessCommitted()
    
  • Nested Statements

    Nested Statements

    I'm trying to run the following sql query, but want to use the query builder:

    SELECT * FROM project_user WHERE project_id IN (SELECT project_id FROM project_user WHERE user_id = 1)
    

    What I have so far:

    innerBuilder := sess.Select("project_id").From("project_user").Where("user_id = ?", userID)
    builder := sess.Select("*").From("project_user").Where("project_id IN ???")
    

    But I don't know how to put the innerBuilder into the outer statement, how can I do that?

  • Unmarshaling joins?

    Unmarshaling joins?

    Nice work! I was just playing with this and couldn't work out how to unmarshal joined tables. For example, in your example:

    sess.Select("*").From("suggestions").
      Join("subdomains", "suggestions.subdomain_id = subdomains.id")
    

    How would you store the query result into a struct using .Load(&suggestions) that contains []subdomain field?

  • Check errors in Load method

    Check errors in Load method

    During iteration with rows.Next() some errors can appear. It will be saved to Rows.lasterr, Next() will return false and iteration will be stopped.

    Normal case is Rows.lasterr == io.EOF, then all data from db has been read. But there are other cases, for example - some transfer problem, then only part of data will be read and Load will not return this error.

    So, it is required to check the latest error with Rows.Err(), it returns any err != nil && err != io.EOF.

  • Support for

    Support for "FOR UPDATE SKIP LOCKED"

    Is there any way to use the select builder to do something like:

    SELECT *
    FROM "mytable"
    WHERE "state" = ? AND "queue" = ?
    ORDER BY "created_on"
    LIMIT 1
    FOR UPDATE SKIP LOCKED
    

    The closest I get is:

    dbr.From("mytable").
    	Where("state = ? and queue = ?", statusQueued, queue).
    	OrderBy("created_on").
    	Limit(1).
    	ForUpdate()
    

    But that misses the last (essential) part SKIP LOCKED

  • [FEATURE] Prewhere statement

    [FEATURE] Prewhere statement

    I think this lib should be Clickhouse first. So I have added key feature of Clickhouse: PREWHERE statement. Example: https://clickhouse.yandex/docs/en/query_language/select/#prewhere-clause

  • ReturnStrings issue with quoted stings

    ReturnStrings issue with quoted stings

    Prepare db:

    create table test
    (
        date  DateTime,
        event String
    ) engine MergeTree() order by date;
    

    Run go code:

    func TestIssue2(t *testing.T) {
    	conn, _ := dbr.Open("clickhouse", "http://user:pass@localhost:8123/db", nil)
    
    	conn.Exec("truncate table test")
    
    	// 4 rows
    	query := `insert into test (date, event) values 
    	   ('2021-01-01 00:00:00', 'click'),
           ('2021-01-01 01:00:00', '"" '),
           ('2021-01-01 02:00:00', '"with quotes" 1'),
           ('2021-01-01 02:00:00', '"with quotes" 2')`
    
    	conn.Exec(query)
    
    	sess := conn.NewSession(nil)
    
    	res, err := sess.Select("event").From("test").ReturnStrings()
    	fmt.Println(err)
    	fmt.Println(len(res))
    	// 2 rows
    }
    

    I expect that it will be 4 rows in result.

  • Wrong placeholder count error on INSERT ... FORMAT CSV

    Wrong placeholder count error on INSERT ... FORMAT CSV

    Here's the case.

    Prepare some db and test if it's working:

    create table test (
         date DateTime,
         event String
    ) engine MergeTree() order by date;
    
    insert into test format CSV
    2021-01-01 00:00:00,click
    2021-01-01 00:00:00,?
    
    select * from test;
    

    Run go code:

    func TestIssue(t *testing.T) {
    	conn, _ := dbr.Open("clickhouse", "http://user:pass@localhost:8123/db", nil)
    
    	query := `insert into test (date, event) format CSV
    2021-01-01 00:00:00,click
    2021-01-01 00:00:00,?`
    
    	_, err := conn.Exec(query)
    	fmt.Println(err) //clickhouse: wrong placeholder count
    
    	query = `insert into test (date, event) format CSV
    "2021-01-01 00:00:00","click"
    "2021-01-01 00:00:00","?"`
    
    	_, err = conn.Exec(query)
    	fmt.Println(err) //clickhouse: wrong placeholder count
    }
    

    I expect to get no errors here, because every query I run is perfectly valid.

  • Add dialect name functions to dialects (clickhouse like)

    Add dialect name functions to dialects (clickhouse like)

    По аналогии со структурой clickhouse добавил в остальные диалекты метод String(), возвращающий название драйвера.

    Существующий пример из диалекта clickhouse

    func (d clickhouse) String() string {
    	return "clickhouse"
    }
    
Go fearless SQL. Sqlvet performs static analysis on raw SQL queries in your Go code base.

Sqlvet Sqlvet performs static analysis on raw SQL queries in your Go code base to surface potential runtime errors at build time. Feature highlights:

Dec 19, 2022
Database Abstraction Layer (dbal) for Go. Support SQL builder and get result easily (now only support mysql)

godbal Database Abstraction Layer (dbal) for go (now only support mysql) Motivation I wanted a DBAL that No ORM、No Reflect、Concurrency Save, support S

Nov 17, 2022
A Go (golang) package that enhances the standard database/sql package by providing powerful data retrieval methods as well as DB-agnostic query building capabilities.

ozzo-dbx Summary Description Requirements Installation Supported Databases Getting Started Connecting to Database Executing Queries Binding Parameters

Dec 31, 2022
LBADD: An experimental, distributed SQL database
LBADD: An experimental, distributed SQL database

LBADD Let's build a distributed database. LBADD is an experimental distributed SQL database, written in Go. The goal of this project is to build a dat

Nov 29, 2022
Mocking your SQL database in Go tests has never been easier.

copyist Mocking your SQL database in Go tests has never been easier. The copyist library automatically records low-level SQL calls made during your te

Dec 19, 2022
Go library for accessing multi-host SQL database installations

hasql hasql provides simple and reliable way to access high-availability database setups with multiple hosts. Status hasql is production-ready and is

Dec 28, 2022
Document-oriented, embedded SQL database

Genji Document-oriented, embedded, SQL database Table of contents Table of contents Introduction Features Installation Usage Using Genji's API Using d

Jan 1, 2023
Fast SQL query builder for Go

sqlf A fast SQL query builder for Go. sqlf statement builder provides a way to: Combine SQL statements from fragments of raw SQL and arguments that ma

Dec 23, 2022
SQL builder and query library for golang

__ _ ___ __ _ _ _ / _` |/ _ \ / _` | | | | | (_| | (_) | (_| | |_| | \__, |\___/ \__, |\__,_| |___/ |_| goqu is an expressive SQL bu

Dec 30, 2022
Type safe SQL builder with code generation and automatic query result data mapping
Type safe SQL builder with code generation and automatic query result data mapping

Jet Jet is a complete solution for efficient and high performance database access, consisting of type-safe SQL builder with code generation and automa

Jan 6, 2023
Type safe SQL query builder and struct mapper for Go

sq (Structured Query) ?? ?? sq is a code-generated, type safe query builder and struct mapper for Go. ?? ?? Documentation • Reference • Examples This

Dec 19, 2022
golang orm and sql builder

gosql gosql is a easy ORM library for Golang. Style: var userList []UserModel err := db.FetchAll(&userList, gosql.Columns("id","name"), gosql.

Dec 22, 2022
This is a SQL Finder, Admin panel finder, and http analyzer that goes basses off requests. Made from 100% golang

This is a SQL Finder, Admin panel finder that goes basses off requests. Made from 100% golang

May 19, 2022
A Golang library for using SQL.

dotsql A Golang library for using SQL. It is not an ORM, it is not a query builder. Dotsql is a library that helps you keep sql files in one place and

Dec 27, 2022
a golang library for sql builder

Gendry gendry is a Go library that helps you operate database. Based on go-sql-driver/mysql, it provides a series of simple but useful tools to prepar

Dec 26, 2022
SQL query builder for Go

GoSQL Query builder with some handy utility functions. Documentation For full documentation see the pkg.go.dev or GitBook. Examples // Open database a

Dec 12, 2022
Write your SQL queries in raw files with all benefits of modern IDEs, use them in an easy way inside your application with all the profit of compile time constants

About qry is a general purpose library for storing your raw database queries in .sql files with all benefits of modern IDEs, instead of strings and co

Dec 25, 2022
💥 A lightweight DSL & ORM which helps you to write SQL in Go.
💥 A lightweight DSL & ORM which helps you to write SQL in Go.

sqlingo is a SQL DSL (a.k.a. SQL Builder or ORM) library in Go. It generates code from the database and lets you write SQL queries in an elegant way.

Jan 2, 2023
Fluent SQL generation for golang

sqrl - fat-free version of squirrel - fluent SQL generator for Go Non thread safe fork of squirrel. The same handy fluffy helper, but with extra lette

Dec 16, 2022