Migration tool for ksqlDB, which uses the ksqldb-go client.

ksqldb-migrate

Migration tool for ksqlDB, which uses the ksqldb-go client.

⚠️ Disclaimer #1: This is a personal project and not supported or endorsed by Confluent.

Installation

go install github.com/thmeitz/ksqldb-migrate

Usage

ksqldb-migrate

Migration File

In further steps I'm using the provided example.

---
up:
  - name: create source connector dogs
    exec: |
      CREATE SOURCE CONNECTOR DOGS WITH ('connector.class'='io.mdrogalis.voluble.VolubleSourceConnector',
      'key.converter'='org.apache.kafka.connect.storage.StringConverter',
      'value.converter'='org.apache.kafka.connect.json.JsonConverter',
      'value.converter.schemas.enable'='false',
      'genkp.dogs.with'='#{Internet.uuid}',
      'genv.dogs.name.with'='#{Dog.name}',
      'genv.dogs.dogsize.with'='#{Dog.size}',
      'genv.dogs.age.with'='#{Dog.age}',
      'topic.dogs.throttle.ms'=1000 
      );
  - name: create the dogs stream
    exec: |
      CREATE STREAM IF NOT EXISTS DOGS (ID STRING KEY,NAME STRING,DOGSIZE STRING, AGE STRING) 
      WITH (KAFKA_TOPIC='dogs', 
      VALUE_FORMAT='JSON', PARTITIONS=1);
  - name: create the DOGS_BY_SIZE table
    exec: |
      CREATE TABLE IF NOT EXISTS DOGS_BY_SIZE AS 
      SELECT DOGSIZE AS DOG_SIZE, COUNT(*) AS DOGS_CT 
      FROM DOGS WINDOW TUMBLING (SIZE 15 MINUTE) 
      GROUP BY DOGSIZE;
down:
  - name: drop table DOGS_BY_SIZE
    exec: DROP TABLE IF EXISTS DOGS_BY_SIZE;
  - name: delete stream DOGS
    exec: DROP STREAM IF EXISTS DOGS;
  - name: drop connector DOGS
    exec: DROP CONNECTOR IF EXISTS DOGS;

Migrate Up

ksqldb-migrate up -f examples/test.yaml

Output

Migrate Down

ksql-migrate down -f examples/test.yaml

Output

Preflight Check

You can make a preflight check if you run up/down.

If there is an error you get following output:

Since the KSQLParser can find more than one error, the errors are in a slice.

The first number gives the row number, the second number the column number followed by the parser message.

Docker Compose

Same docker-compose file as for ksqldb-go.

Build

Only 64 Bit systems are supported.

This command builds the following binaries

  • linux-amd64
  • darwin-amd64
  • windows-amd64
make release github.com/thmeitz/ksqldb-migrate

TODO

  • Better error messages for KSQLParser (ksqldb-go)
  • Check CommandStatus and wait for successfull execution for each step (up/down migrations) (ksql-migrate + ksqldb-go)
  • Static linked binary

License

Apache License Version 2.0

Similar Resources

Type-safe Redis client for Golang

Type-safe Redis client for Golang

Redis client for Golang Join Discord to ask questions. Documentation Reference Examples RealWorld example app Ecosystem Redis Mock. Distributed Locks.

Dec 29, 2022

Go client for AMQP 0.9.1

Go RabbitMQ Client Library This is an AMQP 0.9.1 client with RabbitMQ extensions in Go. Project Maturity This project has been used in production syst

Jan 6, 2023

Interactive client for PostgreSQL and MySQL

Interactive client for PostgreSQL and MySQL

dblab Interactive client for PostgreSQL and MySQL. Overview dblab is a fast and lightweight interactive terminal based UI application for PostgreSQL a

Jan 8, 2023

Cross-platform client for PostgreSQL databases

pgweb Web-based PostgreSQL database browser written in Go. Overview Pgweb is a web-based database browser for PostgreSQL, written in Go and works on O

Dec 30, 2022

[mirror] the database client and tools for the Go vulnerability database

The Go Vulnerability Database golang.org/x/vulndb This repository is a prototype of the Go Vulnerability Database. Read the Draft Design. Neither the

Dec 29, 2022

REST based Redis client built on top of Upstash REST API

An HTTP/REST based Redis client built on top of Upstash REST API.

Jul 31, 2022

A client for TiKV

client-tikv ./tikv-client --pd 127.0.0.1:2379,127.0.0.2:2379,127.0.0.3:2379 usage You can query the value directly according to the key. tikv select

Apr 16, 2022

Client to import measurements to timestream databases.

Timestream DB Client Client to import measurements to timestream databases. Supported Databases/Services AWS Timestream AWS Timestream Run NewTimestre

Jan 11, 2022

Go-clickhouse - ClickHouse client for Go

ClickHouse client for Go 1.18+ This client uses native protocol to communicate w

Jan 9, 2023
Goose database migration tool - fork of https://bitbucket.org/liamstask/goose

goose Goose is a database migration tool. Manage your database schema by creating incremental SQL changes or Go functions. Goals of this fork github.c

Dec 30, 2022
It's a Go console utility for migration from MSSQL to MySQL engine.

A tool for migration the databases to MySQL It's a Go console utility for migration from MSSQL to MySQL engine. The databases should have prepopulated

Jan 4, 2022
🏋️ dbbench is a simple database benchmarking tool which supports several databases and own scripts

dbbench Table of Contents Description Example Installation Supported Databases Usage Custom Scripts Troubeshooting Development Acknowledgements Descri

Dec 30, 2022
sqlx is a library which provides a set of extensions on go's standard database/sql library

sqlx is a library which provides a set of extensions on go's standard database/sql library. The sqlx versions of sql.DB, sql.TX, sql.Stmt, et al. all leave the underlying interfaces untouched, so that their interfaces are a superset on the standard ones. This makes it relatively painless to integrate existing codebases using database/sql with sqlx.

Jan 7, 2023
A reverse proxy for postgres which rewrites queries.

pg-rewrite-proxy A reverse proxy for postgres which rewrites queries. Arbitrary rewriting is supported by supplying an LUA script to the proxy applica

Dec 12, 2022
This package can parse date match expression, which used by ElasticSearch

datemath-parser this package is pure go package, this package can parse date match expression, which used by ElasticSearch. Date Math Definition you c

Jan 8, 2022
Go-postgres - go-postgres library provide NoSQL functionality which can execute queries with pool of connections

GO Postgres go-postgres library provide NoSQL functionality which can execute queries with pool of connections. What is this repository for? Establish

Dec 31, 2021
Cross-platform client for PostgreSQL databases

pgweb Web-based PostgreSQL database browser written in Go. Overview Pgweb is a web-based database browser for PostgreSQL, written in Go and works on O

Dec 30, 2022
Go client for Redis

Redigo Redigo is a Go client for the Redis database. Features A Print-like API with support for all Redis commands. Pipelining, including pipelined tr

Dec 29, 2022