First, thank you for this package and specifically the migration logic (from Bun). That said, I have some concerns about failed migrations being marked as applied. I understand that this allows the rollback logic to be run, but I'm not sure if the desirability of that outweighs the risks.

The major risk I'm worried about is another "up" being performed before the "rollback" and the failed migration being skipped. CI/CD pipelines and developers not aware of this behavior may try to apply the migrations again, like in the case of a timeout or transient error, and be surprised by the behavior. I would argue it violates the principle of least surprise and that the defacto standard is for a following "up" to try applying the failed migration again, not skipping it.

I find the scaleWithCpu define the relationship between batchsize and cpu cores. If i have a 2 core machine, then the batchsize is 2000, every 2000 span start a goroutine and save the data. If i have a 8 core machine, then the batchsize is 8000, every 8000 span start a goroutine and save the data.

I do not understand why it is helpful?

Adds an option that instructs the Migrator to mark migrations applied on success, not on both success and failure.

Something to note, the group is being returned in more of the error paths than before, I think this is consistent with the intent that despite a migration failing some migrations were still applied and that information should be provided to the caller.

Resolves: https://github.com/uptrace/go-clickhouse/issues/25

go: finding module for package github.com/uptrace/go-clickhouse/ch go: found github.com/uptrace/go-clickhouse/ch in github.com/uptrace/go-clickhouse/ch v0.0.0-20220308124651-82a5d8d72ef7 go: downloading github.com/uptrace/go-clickhouse/chdebug v0.0.0-00010101000000-000000000000 github.com/uptrace/go-clickhouse/ch tested by github.com/uptrace/go-clickhouse/ch.test imports github.com/uptrace/go-clickhouse/chdebug: github.com/uptrace/go-clickhouse/[email protected]: invalid version: unknown revision 000000000000

The documentation (http://devdoc.net/database/ClickhouseDocs_19.4.1.3-docs/query_language/select/) says FINAL has to be placed right after the table name, before SAMPLE: SELECT [DISTINCT] expr_list [FROM [db.]table | (subquery) | table_function] [FINAL] [SAMPLE sample_coeff]

At the moment it's appended to the end of the query which results in invalid queries being generated when there are WHERE, ORDER BY or any other clauses used.

type Data struct {
	Str []byte
}

src, _ := hex.DecodeString("5C00CC")
d := &Data{Str: src}

dsn := "clickhouse://localhost:9000/test?sslmode=disable"
db := ch.Connect(ch.WithDSN(dsn), ch.WithAutoCreateDatabase(true))

if err := db.Ping(context.Background()); err != nil {
	panic(err)
}

_, err := db.NewDropTable().IfExists().Model(&Data{}).Exec(context.Background())
if err != nil {
	panic(err)
}

_, err = db.NewCreateTable().IfNotExists().Model(&Data{}).Exec(context.Background())
if err != nil {
	panic(err)
}

_, err = db.NewInsert().Model(d).Exec(context.Background())
if err != nil {
	panic(err)
}

// works
got1 := new(Data)
err = db.NewSelect().Model(got1).
	Where(fmt.Sprintf("str = unhex('%s')", hex.EncodeToString(d.Str))).
	Scan(context.Background())
if err != nil {
	panic(errors.Wrap(err, "[1] cannot select"))
}
fmt.Println("got1:", got1)

// fails
got2 := new(Data)
err = db.NewSelect().Model(got2).
	Where("str = ?", d.Str).
	Scan(context.Background())
if err != nil {
	panic(errors.Wrap(err, "[2] cannot select"))
}
fmt.Println("got2:", got2)

Where clause works well with strings, but fails with sql.ErrNoRows error on byte array.

Here I attach code to reproduce the issue. It panics with [2] cannot select: sql: no rows in result set error.

type Data struct {
	Str []byte
}

d := &Data{Str: []byte("asdf")}

dsn := "clickhouse://localhost:9000/test?sslmode=disable"
db := ch.Connect(ch.WithDSN(dsn), ch.WithAutoCreateDatabase(true))

if err := db.Ping(context.Background()); err != nil {
	panic(err)
}

_, err := db.NewDropTable().IfExists().Model(&Data{}).Exec(context.Background())
if err != nil {
	panic(err)
}

_, err = db.NewCreateTable().IfNotExists().Model(&Data{}).Exec(context.Background())
if err != nil {
	panic(err)
}

_, err = db.NewInsert().Model(d).Exec(context.Background())
if err != nil {
	panic(err)
}

got1 := new(Data)

err = db.NewSelect().Model(got1).
	Where(fmt.Sprintf("str = '%s'", d.Str)).
	Scan(context.Background())
if err != nil {
	panic(errors.Wrap(err, "[1] cannot select"))
}
fmt.Println("got1:", got1)

got2 := new(Data)

err = db.NewSelect().Model(got2).
	Where("str = ?", d.Str).
	Scan(context.Background())
if err != nil {
	panic(errors.Wrap(err, "[2] cannot select"))
}
fmt.Println("got2:", got2)

Bumps go.opentelemetry.io/otel/trace from 1.8.0 to 1.9.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps go.opentelemetry.io/otel/trace from 1.7.0 to 1.8.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

• Fixed no migrations being applied by having it range the unapplied migration, not the empty applied group
• Upping and downing creates an empty group and adds migrations as they are applied (was already doing this for ups, applied to downs as well)
• Renamed the group to applied and unapplied to make the purpose more clear

Related to https://github.com/uptrace/go-clickhouse/issues/30

It looks like the MigrationGroup at https://github.com/uptrace/go-clickhouse/blob/master/chmigrate/migrator.go#L150-L153, is always empty as group.Migrations isn't being set to the unapplied migrations. I assume the range should be on migrations.

given:

type Model struct {
...
  foo []byte `ch:",type:LowCardinality(String)"`
}

.. and then using the fluent table creation + query mechanism(s) of this library, the resulting data (when foo is queried) shows as <[]uint8 value> instead of the actual string-ish data that was intended for insertion.

My Model

type Video struct {
	ch.CHModel `ch:"partition:toYYYYMM(time)"`
	ID         uint64
	Title      string            `ch:""`
	Time       time.Time         `ch:",pk"`
	Meta       []uint64          `ch:"type:Array(UInt64)"`
	Chapter    map[string]string `ch:"type:Map(String,String)"`
}

	video := database.Video{
		ID:    1,
		Title: "Video 1",
		Time:  time.Now(),
		Meta:  []uint64{10, 10, 10, 10},
		Chapter: map[string]string{
			"hello1": "hello",
			"hello":  "hello",
			"title":  "title",
		},
	}
	db.NewInsert().Model(&video).Exec(ctx)

panic: unsupported ClickHouse type="Map(String, String)"

goroutine 1 [running]: github.com/uptrace/go-clickhouse/ch/chschema.goType({0xc0000a4030, 0x13}) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chschema/types.go:307 +0x6ce github.com/uptrace/go-clickhouse/ch/chschema.NewColumnFromCHType({0xc0000a4030, 0x13}, 0xc00009a1e0?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chschema/column.go:54 +0x28 github.com/uptrace/go-clickhouse/ch/chschema.(*Block).Column(0xc00009c280, {0xc00009a1e0, 0x7}, {0xc0000a4030, 0x13}) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chschema/block.go:46 +0xd4 github.com/uptrace/go-clickhouse/ch.(*DB).readBlock.func1() /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/proto.go:515 +0x16e github.com/uptrace/go-clickhouse/ch/chproto.(*Reader).WithCompression(0xc00009c000, 0x1, 0x100e4a7?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chproto/reader.go:44 +0x53 github.com/uptrace/go-clickhouse/ch.(*DB).readBlock(0xc0002c8050, 0xc00009c000, 0xc00009c280, 0x1) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/proto.go:481 +0x8f github.com/uptrace/go-clickhouse/ch.(*DB).readSampleBlock(0x13c3820?, 0xc00009c000) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/proto.go:382 +0x12c github.com/uptrace/go-clickhouse/ch.(*DB)._insert.func1.2(0xc000094000?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:405 +0x25 github.com/uptrace/go-clickhouse/ch/chpool.(*Conn).WithReader(0xc0000a0000, {0x131b410?, 0xc00001e0c0?}, 0x13b2c64?, 0xc00035fb50) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/chpool/conn.go:70 +0x67 github.com/uptrace/go-clickhouse/ch.(*DB)._insert.func1(0xc0000a0000) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:404 +0x12b github.com/uptrace/go-clickhouse/ch.(*DB)._withConn(0xc0002c8050, {0x131b410?, 0xc00001e0c0}, 0xc00035fcf8) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:219 +0x1ec github.com/uptrace/go-clickhouse/ch.(*DB).withConn(0xc0002c8050, {0x131b410?, 0xc00001e0c0?}, 0x12?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:176 +0x28 github.com/uptrace/go-clickhouse/ch.(*DB)._insert(0xc0000922a0?, {0x131b410?, 0xc00001e0c0?}, {0x1000?, 0x1000?}, {0xc0000b5000?, 0x14?}, 0x1000?) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:396 +0x8f github.com/uptrace/go-clickhouse/ch.(*DB).insert(0xc0002c8050, {0x131b410, 0xc00001e0c0}, {0x131afa0, 0xc0000922a0}, {0xc0000b5000, 0x46}, {0xc000092090, 0x5, 0x6}) /Users/sohelmia/go/pkg/mod/github.com/uptrace/[email protected]/ch/db.go:383 +0x145

So, trying this lib but it appears that you missed implementing the conversion from BOOLEAN field to a valid golang bool?

Reproduction is to create a table such as:

CREATE TABLE IF NOT EXISTS test (
    id VARCHAR,
    flag BOOLEAN
)
...

It would be great if we could use structures like this:

type Data struct {
	Int *big.Int `ch:"type:UInt256"`
}

Currently UInt256 type is not supported and *big.Int maps to String type by default.

With compression switched on client parser throws the following exception:

ch: unsupported compression method: 0x51
...
ch: unsupported compression method: 0x00

the hex number at the end of the error looks random

Disabling compression leads to reading timeout issues.

clickhouse/clickhouse-server:22.6

This error doesn't happen consistently. Some queries do get through and are successfully processed.