GitHub's Online Schema Migrations for MySQL

gh-ost

build status downloads release

GitHub's online schema migration for MySQL

gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides pausability, dynamic control/reconfiguration, auditing, and many operational perks.

gh-ost produces a light workload on the master throughout the migration, decoupled from the existing workload on the migrated table.

It has been designed based on years of experience with existing solutions, and changes the paradigm of table migrations.

How?

All existing online-schema-change tools operate in similar manner: they create a ghost table in the likeness of your original table, migrate that table while empty, slowly and incrementally copy data from your original table to the ghost table, meanwhile propagating ongoing changes (any INSERT, DELETE, UPDATE applied to your table) to the ghost table. Finally, at the right time, they replace your original table with the ghost table.

gh-ost uses the same pattern. However it differs from all existing tools by not using triggers. We have recognized the triggers to be the source of many limitations and risks.

Instead, gh-ost uses the binary log stream to capture table changes, and asynchronously applies them onto the ghost table. gh-ost takes upon itself some tasks that other tools leave for the database to perform. As result, gh-ost has greater control over the migration process; can truly suspend it; can truly decouple the migration's write load from the master's workload.

In addition, it offers many operational perks that make it safer, trustworthy and fun to use.

gh-ost general flow

Highlights

  • Build your trust in gh-ost by testing it on replicas. gh-ost will issue same flow as it would have on the master, to migrate a table on a replica, without actually replacing the original table, leaving the replica with two tables you can then compare and satisfy yourself that the tool operates correctly. This is how we continuously test gh-ost in production.
  • True pause: when gh-ost throttles, it truly ceases writes on master: no row copies and no ongoing events processing. By throttling, you return your master to its original workload
  • Dynamic control: you can interactively reconfigure gh-ost, even as migration still runs. You may forcibly initiate throttling.
  • Auditing: you may query gh-ost for status. gh-ost listens on unix socket or TCP.
  • Control over cut-over phase: gh-ost can be instructed to postpone what is probably the most critical step: the swap of tables, until such time that you're comfortably available. No need to worry about ETA being outside office hours.
  • External hooks can couple gh-ost with your particular environment.

Please refer to the docs for more information. No, really, read the docs.

Usage

The cheatsheet has it all. You may be interested in invoking gh-ost in various modes:

  • a noop migration (merely testing that the migration is valid and good to go)
  • a real migration, utilizing a replica (the migration runs on the master; gh-ost figures out identities of servers involved. Required mode if your master uses Statement Based Replication)
  • a real migration, run directly on the master (but gh-ost prefers the former)
  • a real migration on a replica (master untouched)
  • a test migration on a replica, the way for you to build trust with gh-ost's operation.

Our tips:

  • Testing above all, try out --test-on-replica first few times. Better yet, make it continuous. We have multiple replicas where we iterate our entire fleet of production tables, migrating them one by one, checksumming the results, verifying migration is good.
  • For each master migration, first issue a noop
  • Then issue the real thing via --execute.

More tips:

  • Use --exact-rowcount for accurate progress indication
  • Use --postpone-cut-over-flag-file to gain control over cut-over timing
  • Get familiar with the interactive commands

Also see:

What's in a name?

Originally this was named gh-osc: GitHub Online Schema Change, in the likes of Facebook online schema change and pt-online-schema-change.

But then a rare genetic mutation happened, and the c transformed into t. And that sent us down the path of trying to figure out a new acronym. gh-ost (pronounce: Ghost), stands for GitHub's Online Schema Transmogrifier/Translator/Transformer/Transfigurator

License

gh-ost is licensed under the MIT license

gh-ost uses 3rd party libraries, each with their own license. These are found here.

Community

gh-ost is released at a stable state, but with mileage to go. We are open to pull requests. Please first discuss your intentions via Issues.

We develop gh-ost at GitHub and for the community. We may have different priorities than others. From time to time we may suggest a contribution that is not on our immediate roadmap but which may appeal to others.

Please see Coding gh-ost for a guide to getting started developing with gh-ost.

Download/binaries/source

gh-ost is now GA and stable.

gh-ost is available in binary format for Linux and Mac OS/X

Download latest release here

gh-ost is a Go project; it is built with Go 1.14 and above. To build on your own, use either:

  • script/build - this is the same build script used by CI hence the authoritative; artifact is ./bin/gh-ost binary.
  • build.sh for building tar.gz artifacts in /tmp/gh-ost

Generally speaking, master branch is stable, but only releases are to be used in production.

Authors

gh-ost is designed, authored, reviewed and tested by the database infrastructure team at GitHub:

Owner
GitHub
How people build software.
GitHub
https://github.com/github/gh-ost
Comments
  • Force index in unique key range queries

    Force index in unique key range queries

    Description

    This PR forces the unique-key index to be used when gathering the min/max range of the unique key

    This is to avoid possibilities of the optimizer picking a different index than we planned, discussed in more detail here. cc @morgo / @cyrinux

    In case this PR introduced Go code changes:

    • [x] contributed code is using same conventions as original code
    • [x] script/cibuild returns with no formatting errors, build errors or unit test errors.
  • feat(#1235): add an optimizer switch flag to migration context

    feat(#1235): add an optimizer switch flag to migration context

    Description

    This add a optimizer_switch string flag.

    --optimizer-switch="prefer_ordering_index=on"

    I don't check the value, easier to maintains, this will just fail if its a unknown value, depending the mysql/maria/percona version.

    Related issue: https://github.com/github/gh-ost/issues/1235

  • add the possibility to toggle the optimizer switch during the migration session

    add the possibility to toggle the optimizer switch during the migration session

    Hi,

    We realized we have for some databases the need to be able to set the migration session optimizer switch, for example set prefer_ordering_index=on

    https://dev.mysql.com/doc/refman/8.0/en/switchable-optimizations.html

    I will do a PR for this.

  • The table cut failed and caused the slave library to block

    The table cut failed and caused the slave library to block

    Excuse me, I would like to ask a question. When using gh-ost to execute ddl yesterday, an error occurred in the main library, which caused the table to fail to switch successfully. But there are two errors, and I am not sure which error caused the table to fail to switch successfully.

    2022/12/19 16:33:11 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 root false false } 2022/12/19 16:33:11 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000196, 1004574218) 2022/12/19 16:33:11 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:33:11 binlogsyncer.go:573: [info] rotate to (mysql-bin.000196, 1004574218) Migrating customer3.customer_company; Ghost table is customer3._customer_company_gho Migrating rdsdbn3-sirius-customer-online:3306; inspecting rdsdbn3-sirius-customer-online:3306; executing on rdsdbn3-sirius-customer-online Migration started at Mon Dec 19 16:33:11 +0800 2022 chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: Threads_running=100; critical-load: Threads_running=500; nice-ratio: 0.000000 throttle-additional-flag-file: /tmp/gh-ost.throttle Serving on unix socket: /tmp/gh-ost.customer3.customer_company.sock Serving on TCP port: 13316 Copy: 0/821093 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: mysql-bin.000196:1004589539; State: migrating; ETA: N/A Copy: 0/821093 0.0%; Applied: 0; Backlog: 0/1000; Time: 1s(total), 1s(copy); streamer: mysql-bin.000196:1004965536; State: migrating; ETA: N/A Copy: 9000/821093 1.1%; Applied: 0; Backlog: 0/1000; Time: 2s(total), 2s(copy); streamer: mysql-bin.000196:1007404907; State: migrating; ETA: 3m0s Copy: 18000/821093 2.2%; Applied: 0; Backlog: 0/1000; Time: 3s(total), 3s(copy); streamer: mysql-bin.000196:1010115047; State: migrating; ETA: 2m13s ... ... ... 2022/12/19 16:35:14 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:35:17 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:17 ERROR 2022-12-19 16:35:17 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:19 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:19 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:19 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:19 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:35:21 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:35:21 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:21 ERROR 2022-12-19 16:35:21 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:26 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:26 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:26 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:26 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m15s(total), 2m0s(copy); streamer: mysql-bin.000197:3096692; State: migrating; ETA: due 2022-12-19 16:35:27 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:27 ERROR 2022-12-19 16:35:27 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:28 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m20s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:35:31 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:31 ERROR 2022-12-19 16:35:31 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:33 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:33 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:33 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:33 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:35:35 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21536965; State: migrating; ETA: due 2022-12-19 16:35:37 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:37 ERROR 2022-12-19 16:35:37 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:40 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:40 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:40 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:40 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m30s(total), 2m0s(copy); streamer: mysql-bin.000197:10420122; State: migrating; ETA: due 2022/12/19 16:35:43 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:35:43 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:43 ERROR 2022-12-19 16:35:43 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:35:48 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:48 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:48 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:48 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:35:50 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:50 ERROR 2022-12-19 16:35:50 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:50 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m40s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:35:55 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:55 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:55 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:55 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:35:56 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:56 ERROR 2022-12-19 16:35:56 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m45s(total), 2m0s(copy); streamer: mysql-bin.000197:9353085; State: migrating; ETA: due 2022/12/19 16:35:58 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:36:02 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:02 ERROR 2022-12-19 16:36:02 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:03 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:03 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:03 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:03 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:36:05 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m55s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:36:08 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:08 ERROR 2022-12-19 16:36:08 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:10 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:10 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:10 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:10 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m0s(total), 2m0s(copy); streamer: mysql-bin.000197:17516759; State: migrating; ETA: due 2022/12/19 16:36:12 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:36:12 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:12 ERROR 2022-12-19 16:36:12 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:16 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:16 ERROR 2022-12-19 16:36:16 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21608867; State: migrating; ETA: due 2022/12/19 16:36:17 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:17 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:17 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:17 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:36:18 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21584851; State: migrating; ETA: due 2022-12-19 16:36:22 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:22 ERROR 2022-12-19 16:36:22 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:23 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:23 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:23 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:23 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:36:25 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:36:26 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:26 ERROR 2022-12-19 16:36:26 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m15s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:36:30 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:30 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:30 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:30 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m20s(total), 2m0s(copy); streamer: mysql-bin.000197:12833146; State: migrating; ETA: due 2022/12/19 16:36:32 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:36:32 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:32 ERROR 2022-12-19 16:36:32 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:36:37 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:37 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:37 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:37 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:36:38 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:38 ERROR 2022-12-19 16:36:38 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:38 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m30s(total), 2m0s(copy); streamer: mysql-bin.000197:21569140; State: migrating; ETA: due 2022/12/19 16:36:43 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:43 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:43 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:43 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:36:44 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:44 ERROR 2022-12-19 16:36:44 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:45 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:36:50 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:50 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:50 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:50 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:36:51 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:51 ERROR 2022-12-19 16:36:51 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m40s(total), 2m0s(copy); streamer: mysql-bin.000197:18875201; State: migrating; ETA: due 2022/12/19 16:36:51 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m45s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:36:56 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:56 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:56 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:56 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:36:57 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:57 ERROR 2022-12-19 16:36:57 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:58 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:37:03 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:03 ERROR 2022-12-19 16:37:03 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:03 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:03 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:03 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:03 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:05 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m55s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:37:09 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:09 ERROR 2022-12-19 16:37:09 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:10 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:10 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:10 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:10 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m0s(total), 2m0s(copy); streamer: mysql-bin.000197:15550736; State: migrating; ETA: due 2022/12/19 16:37:12 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:37:16 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:16 ERROR 2022-12-19 16:37:16 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:37:17 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:17 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:17 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:17 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:18 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:37:22 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:22 ERROR 2022-12-19 16:37:22 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:23 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:23 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:23 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:23 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:25 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m15s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:37:28 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:28 ERROR 2022-12-19 16:37:28 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:30 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:30 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:30 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:30 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m20s(total), 2m0s(copy); streamer: mysql-bin.000197:8319580; State: migrating; ETA: due 2022/12/19 16:37:32 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:37:34 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:34 ERROR 2022-12-19 16:37:34 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:37:37 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:37 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:37 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:37 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:39 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:37:40 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:40 ERROR 2022-12-19 16:37:40 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m30s(total), 2m0s(copy); streamer: mysql-bin.000197:21624860; State: migrating; ETA: due 2022/12/19 16:37:44 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:44 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:44 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:44 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:46 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21584851; State: migrating; ETA: due 2022-12-19 16:37:47 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:47 ERROR 2022-12-19 16:37:47 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:51 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:51 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:51 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:51 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m40s(total), 2m0s(copy); streamer: mysql-bin.000197:3346793; State: migrating; ETA: due 2022-12-19 16:37:53 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:53 ERROR 2022-12-19 16:37:53 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:53 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m45s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:37:58 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:58 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:58 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:58 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:37:59 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:59 ERROR 2022-12-19 16:37:59 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:00 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:38:05 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:05 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:05 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:05 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:38:05 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:05 ERROR 2022-12-19 16:38:05 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m55s(total), 2m0s(copy); streamer: mysql-bin.000197:8519807; State: migrating; ETA: due 2022/12/19 16:38:07 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m0s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:38:12 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:12 ERROR 2022-12-19 16:38:12 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:12 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:12 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:12 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:12 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:38:16 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:38:18 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:18 ERROR 2022-12-19 16:38:18 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:21 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:21 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:21 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:21 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m10s(total), 2m0s(copy); streamer: mysql-bin.000197:2841618; State: migrating; ETA: due 2022/12/19 16:38:24 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:38:25 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:25 ERROR 2022-12-19 16:38:25 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m15s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:38:29 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:29 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:29 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:29 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:38:31 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:38:31 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:31 ERROR 2022-12-19 16:38:31 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m20s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:38:36 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:36 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:36 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:36 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m25s(total), 2m0s(copy); streamer: mysql-bin.000197:4548675; State: migrating; ETA: due 2022-12-19 16:38:37 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:37 ERROR 2022-12-19 16:38:37 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:37 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m30s(total), 2m0s(copy); streamer: mysql-bin.000197:21569140; State: migrating; ETA: due 2022/12/19 16:38:42 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:42 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:42 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:42 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:38:43 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:43 ERROR 2022-12-19 16:38:43 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:44 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21561394; State: migrating; ETA: due 2022/12/19 16:38:49 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:49 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:49 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:49 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:38:49 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:49 ERROR 2022-12-19 16:38:49 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m40s(total), 2m0s(copy); streamer: mysql-bin.000197:20796315; State: migrating; ETA: due 2022/12/19 16:38:51 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:38:55 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:55 ERROR 2022-12-19 16:38:55 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m45s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:38:56 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:56 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:56 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:56 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:38:58 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:39:02 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:02 ERROR 2022-12-19 16:39:02 ERROR Timeout while waiting for events up to lock 2022/12/19 16:39:03 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:03 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:03 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:03 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:39:05 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m55s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:39:08 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:08 ERROR 2022-12-19 16:39:08 ERROR Timeout while waiting for events up to lock 2022/12/19 16:39:10 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:10 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:10 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:10 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m0s(total), 2m0s(copy); streamer: mysql-bin.000197:15982497; State: migrating; ETA: due 2022/12/19 16:39:12 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:14 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:14 ERROR 2022-12-19 16:39:14 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21616782; State: migrating; ETA: due 2022/12/19 16:39:17 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:17 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:17 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:17 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:39:19 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:20 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:20 ERROR 2022-12-19 16:39:20 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21569140; State: migrating; ETA: due 2022/12/19 16:39:24 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:24 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:24 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:24 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:39:25 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:26 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:26 ERROR 2022-12-19 16:39:26 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m15s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:30 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:30 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:30 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:30 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m20s(total), 2m0s(copy); streamer: mysql-bin.000197:15536690; State: migrating; ETA: due 2022/12/19 16:39:32 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:32 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:32 ERROR 2022-12-19 16:39:32 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:37 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:37 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:37 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:37 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:39:38 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:38 ERROR 2022-12-19 16:39:38 ERROR Timeout while waiting for events up to lock 2022/12/19 16:39:38 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m30s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:43 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:43 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:43 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:43 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:39:45 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction 2022-12-19 16:39:45 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction 2022/12/19 16:39:45 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:50 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:50 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:50 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:50 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m40s(total), 2m0s(copy); streamer: mysql-bin.000197:7727416; State: migrating; ETA: due 2022/12/19 16:39:52 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:53 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:53 ERROR 2022-12-19 16:39:53 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m45s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:57 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:57 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:57 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:57 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:39:59 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:59 ERROR 2022-12-19 16:39:59 ERROR Timeout while waiting for events up to lock 2022/12/19 16:39:59 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:40:04 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:04 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:04 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:04 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:40:05 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:05 ERROR 2022-12-19 16:40:05 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m55s(total), 2m0s(copy); streamer: mysql-bin.000197:19093954; State: migrating; ETA: due 2022/12/19 16:40:07 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:40:11 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:11 ERROR 2022-12-19 16:40:11 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m0s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:40:12 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:12 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:12 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:12 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:40:13 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:40:17 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:17 ERROR 2022-12-19 16:40:17 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:18 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:18 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:18 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:18 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:40:20 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:40:23 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:23 ERROR 2022-12-19 16:40:23 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:25 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:25 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:25 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:25 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m15s(total), 2m0s(copy); streamer: mysql-bin.000197:6506231; State: migrating; ETA: due 2022-12-19 16:40:27 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:27 ERROR 2022-12-19 16:40:27 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:29 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m20s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:40:31 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:31 ERROR 2022-12-19 16:40:31 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:34 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:34 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:34 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:34 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:40:35 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:35 ERROR 2022-12-19 16:40:35 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:36 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21553308; State: migrating; ETA: due 2022-12-19 16:40:39 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:39 ERROR 2022-12-19 16:40:39 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:41 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:41 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:41 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:41 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m30s(total), 2m0s(copy); streamer: mysql-bin.000197:8687216; State: migrating; ETA: due 2022/12/19 16:40:42 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:40:43 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:43 ERROR 2022-12-19 16:40:43 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:40:47 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:47 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:47 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:47 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:40:47 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:47 ERROR 2022-12-19 16:40:47 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:49 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m40s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:40:51 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:51 ERROR 2022-12-19 16:40:51 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:54 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:54 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:54 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:54 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:40:55 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:55 ERROR 2022-12-19 16:40:55 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:56 FATAL 2022-12-19 16:40:55 ERROR Timeout while waiting for events up to lock

    There are two errors here: 1、 [error] close sync with err: data len 310 < expected 2179 The trigger reason for this error is not detailed，i don't know what caused it.

    2、 ERROR Timeout while waiting for events up to lock Here I think that due to the existence of large transactions, multiple acquisitions of metadata locks failed, resulting in timeouts for acquiring metadata locks.

    In addition, the execution failure on the main library will also cause the replication of the slave library to be stuck, and the drop table and multi-threaded replication will cause lock conflicts. image

  • feat(binlogsyncer): support set binlogsyncer maxreconnectattempts

    feat(binlogsyncer): support set binlogsyncer maxreconnectattempts

    A Pull Request should be associated with an Issue.

    We wish to have discussions in Issues. A single issue may be targeted by multiple PRs. If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues, and potentially we'll be able to point development in a particular direction.

    Related issue: https://github.com/github/gh-ost/issues/0123456789

    Further notes in https://github.com/github/gh-ost/blob/master/.github/CONTRIBUTING.md Thank you! We are open to PRs, but please understand if for technical reasons we are unable to accept each and any PR

    Description

    This PR [briefly explain what it does]

    In case this PR introduced Go code changes:

    • [ ] contributed code is using same conventions as original code
    • [ ] script/cibuild returns with no formatting errors, build errors or unit test errors.
  • use last second copy rate to project ETA

    use last second copy rate to project ETA

    A Pull Request should be associated with an Issue.

    We wish to have discussions in Issues. A single issue may be targeted by multiple PRs. If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues, and potentially we'll be able to point development in a particular direction.

    Related issue: https://github.com/github/gh-ost/issues/1204

    Further notes in https://github.com/github/gh-ost/blob/master/.github/CONTRIBUTING.md Thank you! We are open to PRs, but please understand if for technical reasons we are unable to accept each and any PR

    Description

    Currently, the ETA that gh-ost uses is based on a projection that averages over the whole run time. Because (with a b+tree) the insert rate will slow down as the table gets larger, this can lead to inaccurate projections.

    This uses a different approach, which is to project from the number of rows copied in the last second. It has the downside that the estimate can tend to "jump around" a bit, but sometimes this is helpful. I played around with changing the sample average to 10s etc, but it looks like 1s seems to be long enough to balance estimate stability.

    Here is a sample output showing the time bouncing around a little:

    ~/go/src/github.com/morgo/gh-ost/go/cmd/gh-ost more-accurate-eta go run . \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--user="msandbox" \
--password="msandbox" \
--host="127.0.0.1" \
--port=8031 \
--allow-on-master \
--database="test" \
--table="customer" \
--initially-drop-old-table \
--ok-to-drop-table \
--execute  --initially-drop-ghost-table --alter="engine=innodb"
[2022/12/14 14:28:21] [info] binlogsyncer.go:148 create BinlogSyncer with config {99999 mysql 127.0.0.1 8031 msandbox    false false <nil> false UTC true 0 0s 0s 0 false false 0 <nil>}
[2022/12/14 14:28:21] [info] binlogsyncer.go:374 begin to sync binlog from position (binlog.000199, 157)
[2022/12/14 14:28:21] [info] binlogsyncer.go:791 rotate to (binlog.000199, 157)
# Migrating `test`.`customer`; Ghost table is `test`.`_customer_gho`
# Migrating mtocker-macbookpro.local:8031; inspecting mtocker-macbookpro.local:8031; executing on mtocker-macbookpro.local
# Migration started at Wed Dec 14 14:28:21 -0700 2022
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: Threads_running=25; critical-load: Threads_running=1000; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# Serving on unix socket: /tmp/gh-ost.test.customer.sock
Copy: 0/2881784 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: binlog.000199:3566; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: N/A
Copy: 0/2881784 0.0%; Applied: 0; Backlog: 0/1000; Time: 1s(total), 1s(copy); streamer: binlog.000199:8040; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: N/A
Copy: 84000/2881784 2.9%; Applied: 0; Backlog: 0/1000; Time: 2s(total), 2s(copy); streamer: binlog.000199:47281787; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 1m6s
Copy: 168000/2881784 5.8%; Applied: 0; Backlog: 0/1000; Time: 3s(total), 3s(copy); streamer: binlog.000199:94566987; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 245000/2881784 8.5%; Applied: 0; Backlog: 0/1000; Time: 4s(total), 4s(copy); streamer: binlog.000199:137949978; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 34s
Copy: 323000/2881784 11.2%; Applied: 0; Backlog: 0/1000; Time: 5s(total), 5s(copy); streamer: binlog.000199:181313206; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 402000/2881784 13.9%; Applied: 0; Backlog: 0/1000; Time: 6s(total), 6s(copy); streamer: binlog.000199:226333018; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 31s
Copy: 476000/2881784 16.5%; Applied: 0; Backlog: 0/1000; Time: 7s(total), 7s(copy); streamer: binlog.000199:268011561; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 555000/2881784 19.3%; Applied: 0; Backlog: 0/1000; Time: 8s(total), 8s(copy); streamer: binlog.000199:312501660; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 29s
Copy: 615000/2881784 21.3%; Applied: 0; Backlog: 0/1000; Time: 9s(total), 9s(copy); streamer: binlog.000199:346275029; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 37s
Copy: 669000/2881784 23.2%; Applied: 0; Backlog: 0/1000; Time: 10s(total), 10s(copy); streamer: binlog.000199:376674938; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 40s
Copy: 726000/2881784 25.2%; Applied: 0; Backlog: 0/1000; Time: 11s(total), 11s(copy); streamer: binlog.000199:408728243; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 37s
Copy: 783000/2881784 27.2%; Applied: 0; Backlog: 0/1000; Time: 12s(total), 12s(copy); streamer: binlog.000199:440842291; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 36s
Copy: 846000/2881784 29.4%; Applied: 0; Backlog: 0/1000; Time: 13s(total), 13s(copy); streamer: binlog.000199:476296347; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 910000/2881784 31.6%; Applied: 0; Backlog: 0/1000; Time: 14s(total), 14s(copy); streamer: binlog.000199:512338908; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 30s
Copy: 969000/2881784 33.6%; Applied: 0; Backlog: 0/1000; Time: 15s(total), 15s(copy); streamer: binlog.000199:545572110; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 1026000/2881784 35.6%; Applied: 0; Backlog: 0/1000; Time: 16s(total), 16s(copy); streamer: binlog.000199:577675650; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 1087000/2881784 37.7%; Applied: 0; Backlog: 0/1000; Time: 17s(total), 17s(copy); streamer: binlog.000199:612023226; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 29s
Copy: 1148000/2881784 39.8%; Applied: 0; Backlog: 0/1000; Time: 18s(total), 18s(copy); streamer: binlog.000199:646342977; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 28s
Copy: 1204000/2881784 41.8%; Applied: 0; Backlog: 0/1000; Time: 19s(total), 19s(copy); streamer: binlog.000199:677881385; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 29s
Copy: 1265000/2881784 43.9%; Applied: 0; Backlog: 0/1000; Time: 20s(total), 20s(copy); streamer: binlog.000199:712235510; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 26s
Copy: 1335000/2881784 46.3%; Applied: 0; Backlog: 0/1000; Time: 21s(total), 21s(copy); streamer: binlog.000199:751664902; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 22s
Copy: 1403000/2881784 48.7%; Applied: 0; Backlog: 0/1000; Time: 22s(total), 22s(copy); streamer: binlog.000199:789461333; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 21s
Copy: 1464000/2881784 50.8%; Applied: 0; Backlog: 0/1000; Time: 23s(total), 23s(copy); streamer: binlog.000199:824289714; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 23s
Copy: 1527000/2881784 53.0%; Applied: 0; Backlog: 0/1000; Time: 24s(total), 24s(copy); streamer: binlog.000199:859468766; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 21s
Copy: 1592000/2881784 55.2%; Applied: 0; Backlog: 0/1000; Time: 25s(total), 25s(copy); streamer: binlog.000199:896372430; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 19s
Copy: 1662000/2881784 57.7%; Applied: 0; Backlog: 0/1000; Time: 26s(total), 26s(copy); streamer: binlog.000199:935798318; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 17s
Copy: 1725000/2881784 59.9%; Applied: 0; Backlog: 0/1000; Time: 27s(total), 27s(copy); streamer: binlog.000199:971271951; Lag: 0.11s, HeartbeatLag: 0.11s, State: migrating; ETA: 18s
Copy: 1793000/2881784 62.2%; Applied: 0; Backlog: 0/1000; Time: 28s(total), 28s(copy); streamer: binlog.000199:1009545478; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 16s
Copy: 1856000/2881784 64.4%; Applied: 0; Backlog: 0/1000; Time: 29s(total), 29s(copy); streamer: binlog.000199:1044945554; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 16s
[2022/12/14 14:28:51] [info] binlogsyncer.go:791 rotate to (binlog.000200, 4)
[2022/12/14 14:28:51] [info] binlogsyncer.go:791 rotate to (binlog.000200, 4)
Copy: 1920000/2881784 66.6%; Applied: 0; Backlog: 0/1000; Time: 30s(total), 30s(copy); streamer: binlog.000200:6753737; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 15s
Copy: 1977000/2881784 68.6%; Applied: 0; Backlog: 0/1000; Time: 31s(total), 31s(copy); streamer: binlog.000200:38877488; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 15s
Copy: 2032000/2881784 70.5%; Applied: 0; Backlog: 0/1000; Time: 32s(total), 32s(copy); streamer: binlog.000200:69863585; Lag: 0.11s, HeartbeatLag: 0.11s, State: migrating; ETA: 15s
Copy: 2093000/2881784 72.6%; Applied: 0; Backlog: 0/1000; Time: 33s(total), 33s(copy); streamer: binlog.000200:104128743; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 12s
Copy: 2158000/2881784 74.9%; Applied: 0; Backlog: 0/1000; Time: 34s(total), 34s(copy); streamer: binlog.000200:140822753; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 11s
Copy: 2220000/2881784 77.0%; Applied: 0; Backlog: 0/1000; Time: 35s(total), 35s(copy); streamer: binlog.000200:175718354; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 10s
Copy: 2280000/2881784 79.1%; Applied: 0; Backlog: 0/1000; Time: 36s(total), 36s(copy); streamer: binlog.000200:209458784; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 10s
Copy: 2342000/2881784 81.3%; Applied: 0; Backlog: 0/1000; Time: 37s(total), 37s(copy); streamer: binlog.000200:244384714; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 8s
Copy: 2405000/2881784 83.5%; Applied: 0; Backlog: 0/1000; Time: 38s(total), 38s(copy); streamer: binlog.000200:279865282; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 7s
Copy: 2469000/2881784 85.7%; Applied: 0; Backlog: 0/1000; Time: 39s(total), 39s(copy); streamer: binlog.000200:315425854; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 6s
Copy: 2529000/2881784 87.8%; Applied: 0; Backlog: 0/1000; Time: 40s(total), 40s(copy); streamer: binlog.000200:349673270; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 5s
Copy: 2598000/2881784 90.2%; Applied: 0; Backlog: 0/1000; Time: 41s(total), 41s(copy); streamer: binlog.000200:388534212; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 4s
Copy: 2661000/2881784 92.3%; Applied: 0; Backlog: 0/1000; Time: 42s(total), 42s(copy); streamer: binlog.000200:423824648; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 3s
Copy: 2718000/2881784 94.3%; Applied: 0; Backlog: 0/1000; Time: 43s(total), 43s(copy); streamer: binlog.000200:456090786; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 2s
Copy: 2775000/2881784 96.3%; Applied: 0; Backlog: 0/1000; Time: 44s(total), 44s(copy); streamer: binlog.000200:488195145; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 1s
Copy: 2833000/2881784 98.3%; Applied: 0; Backlog: 0/1000; Time: 45s(total), 45s(copy); streamer: binlog.000200:520841157; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: due
Copy: 2894000/2881784 100.4%; Applied: 0; Backlog: 0/1000; Time: 46s(total), 46s(copy); streamer: binlog.000200:555202612; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: due
Copy: 2952000/2881784 102.4%; Applied: 0; Backlog: 0/1000; Time: 47s(total), 47s(copy); streamer: binlog.000200:587420664; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: due
Copy: 3000000/3000000 100.0%; Applied: 0; Backlog: 0/1000; Time: 47s(total), 47s(copy); streamer: binlog.000200:614436979; Lag: 0.01s, HeartbeatLag: 0.03s, State: migrating; ETA: due
Copy: 3000000/3000000 100.0%; Applied: 0; Backlog: 1/1000; Time: 48s(total), 47s(copy); streamer: binlog.000200:614902121; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: due
# Migrating `test`.`customer`; Ghost table is `test`.`_customer_gho`
# Migrating mtocker-macbookpro.local:8031; inspecting mtocker-macbookpro.local:8031; executing on mtocker-macbookpro.local
# Migration started at Wed Dec 14 14:28:21 -0700 2022
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: Threads_running=25; critical-load: Threads_running=1000; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# Serving on unix socket: /tmp/gh-ost.test.customer.sock
Copy: 3000000/3000000 100.0%; Applied: 0; Backlog: 0/1000; Time: 48s(total), 47s(copy); streamer: binlog.000200:614905425; Lag: 0.01s, HeartbeatLag: 0.03s, State: migrating; ETA: due
[2022/12/14 14:29:10] [info] binlogsyncer.go:180 syncer is closing...
[2022/12/14 14:29:10] [info] binlogsyncer.go:864 kill last connection id 11
[2022/12/14 14:29:10] [info] binlogsyncer.go:210 syncer is closed
# Done

    In case this PR introduced Go code changes:

    • [x] contributed code is using same conventions as original code
    • [x] script/cibuild returns with no formatting errors, build errors or unit test errors.
Related tags
Database tools mysql schema-migrations
Database migrations. CLI and Golang library.

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

Jan 9, 2023
Database migrations written in Go

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

Oct 3, 2021
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置，参数n为秒数，查询结

Nov 15, 2022
Manage Schema for KubeDB managed Databases

schema-manager Manage Schema for KubeDB managed Databases Installation To install KubeDB, please follow the guide here. Using KubeDB Want to learn how

Feb 19, 2022
Go library that stores data in Redis with SQL-like schema

Go library that stores data in Redis with SQL-like schema. The goal of this library is we can store data in Redis with table form.

Mar 14, 2022
Web-based, zero-config, dependency-free database schema change and version control tool for teams
 Web-based, zero-config, dependency-free database schema change and version control tool for teams

Live Demo • Install • Help • Development • Design Doc Bytebase is a web-based, zero-config, dependency-free database schema change and version control

Jan 1, 2023
a powerful mysql toolset with Go
 a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

Dec 28, 2022
Sync MySQL data into elasticsearch
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

Dec 30, 2022
A high-performance MySQL proxy

kingshard 中文主页 Overview kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read

Dec 30, 2022
Golang MySql binary log replication listener

Go MySql binary log replication listener Pure Go Implementation of MySQL replication protocol. This allow you to receive event like insert, update, de

Oct 25, 2022
MySQL replication topology management and HA
 MySQL replication topology management and HA

orchestrator [Documentation] orchestrator is a MySQL high availability and replication management tool, runs as a service and provides command line ac

Jan 4, 2023
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Jan 3, 2023
db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

Nov 17, 2022
一个使 mysql，pgsql 数据库表自动生成 go struct 的工具

db2go 一个使 mysql、pgsql 数据库表自动生成 go struct 的工具 快速使用 将项目放入到GOPATH/src目录下

Nov 25, 2022
🐳 A most popular sql audit platform for mysql
🐳 A most popular sql audit platform for mysql

?? A most popular sql audit platform for mysql

Jan 6, 2023
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

Nov 9, 2022
GoRose(go orm), a mini database ORM for golang, which inspired by the famous php framwork laravle's eloquent. It will be friendly for php developer and python or ruby developer. Currently provides six major database drivers: mysql,sqlite3,postgres,oracle,mssql, Clickhouse.
 GoRose(go orm), a mini database ORM for golang, which inspired by the famous php framwork laravle's eloquent. It will be friendly for php developer and python or ruby developer. Currently provides six major database drivers: mysql,sqlite3,postgres,oracle,mssql, Clickhouse.

GoRose ORM _______ ______ .______ ______ _______. _______ / _____| / __ \ | _ \ / __ \ / || ____| |

Dec 24, 2022
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Jan 4, 2023
Gaea is a mysql proxy, it's developed by xiaomi b2c-dev team.
 Gaea is a mysql proxy, it's developed by xiaomi b2c-dev team.

简介 Gaea是小米中国区电商研发部研发的基于mysql协议的数据库中间件，目前在小米商城大陆和海外得到广泛使用，包括订单、社区、活动等多个业务。Gaea支持分库分表、sql路由、读写分离等基本特性，更多详细功能可以参照下面的功能列表。其中分库分表方案兼容了mycat和kingshard两个项目的路

Dec 30, 2022