Feature Request

Is your feature request related to a problem? Please describe:

Currently when clustered index is enabled, the _tidb_rowid column is missing (?). Dumpling will use _tidb_rowid to parallel dumping tables.

Describe the feature you'd like:

Support clustered index.

Reference: https://github.com/pingcap/tidb/pull/17044/files

Describe alternatives you've considered:

No alternatives.

Teachability, Documentation, Adoption, Optimization:

[2020/03/01 21:07:14.710 +08:00] [WARN] [session.go:1123] ["compile SQL failed"] [conn=51] [error="[planner:1054]Unknown column '_tidb_rowid' in 'field list'"] [SQL="SELECT _tidb_rowid from INFORMATION_SCHEMA.INSPECTION_RESULT LIMIT 0"]

Seem table INSPECTION_RESULT has no _tidb_rowid

What problem does this PR solve?

pingcap/check is no longer under maintenance, migrate to testify.

This PR can be somewhat regarded as a subtask of https://github.com/pingcap/tidb/issues/26022

This closes #346 .

Release note

• No release note

What problem does this PR solve?

None

What is changed and how it works?

None

Check List

Tests

• No code

Side effects

None

Related changes

None

Release note

• No release note

Question

I have read the docs and know the GCS stuff is experimental but I wanted to see if I could replace mysqldump. I set the --gcs.endpoint URL and

A) it never wrote the gcs. (did not error either) B) Is it possible to write straight to GCS? I don't have enough space on this host to write to disk.

Thanks, Jeff

What problem does this PR solve?

#8 Support S3

What is changed and how it works?

Switch to using VFS which supports local files, S3 and GCS. For now this only supports S3 but it should be easy to add support to e.g. GCS.

You simply prefix the output directory with "s3://bucket" e.g.:

dumpling -o s3://mybycket/dir

For now all config is passed in using the standard AWS environment variables.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below) Will test by running dumpling against our staging databases.

Side effects

• Introduces a new dependency on github.com/c2fo/vfs

Related changes

• Need to update the documentation

Release note

• support writing directly to S3

When Dumpling dumps large data it needs to set TiDB snapshot at a ts. When the dumping time is too long tikv may gc this TiDB snapshot ts and Dumpling will fail to dump, exits with an error.

To avoid this problem, users need to set tikv_gc_life_time longer, which is explained in the end of https://pingcap.com/docs-cn/stable/export-or-backup-using-dumpling. However, it's not convenient. The users need to run

update mysql.tidb set VARIABLE_VALUE = '720h' where VARIABLE_NAME = 'tikv_gc_life_time';

before dumping and run

update mysql.tidb set VARIABLE_VALUE = '10m' where VARIABLE_NAME = 'tikv_gc_life_time';

after dumping.

But, Dumpling shouldn't change this variable because dumping process may fail in the half.

In v4.0.0, PD has a new interface UpdateServiceGCSafePoint which allows service to set TiKV GC safe point with a ttl, which is very suitable for dumpling's scene. What's more, TiCDC has already used this interface in their service.

So, in summary, I think dumpling can also use this feature to make sure dumped ts is not "GCed".

What problem does this PR solve?

this error is hard to reproduce, so I added a 10 minutes sleep before any dump indeed occurs. The error may be slightly different but I think the root cause is same . Here's the log

[2021/06/03 19:20:30.262 +08:00] [INFO] [gc_worker.go:1531] ["[gc worker] sent safe point to PD"] [uuid=5e7451a0ff80001] ["safe point"=425385803120640001]
...
[2021/06/03 19:22:21.578 +08:00] [INFO] [set.go:312] ["load snapshot info schema"] [conn=5813] [SnapshotTS=425385803120640000]
...
[2021/06/03 19:22:21.579 +08:00] [INFO] [conn.go:812] ["command dispatched failed"] [conn=5813] [connInfo="id:5813, addr:127.0.0.1:62687 status:10, collation:utf8mb4_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="SET tidb_snapshot='2021-06-03 19:01:00'"] [txn_mode=PESSIMISTIC] [err="[tikv:9006]GC life time is shorter than transaction duration, transaction starts at 2021-06-03 19:01:00 +0800 CST, GC safe point is 2021-06-03 19:01:00 +0800 CST\ngithub.com/pingcap/errors.AddStack\n\t/Users/pingcap/gopkg/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/Users/pingcap/gopkg/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/tidb/store/tikv.(*KVStore).CheckVisibility\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/store/tikv/kv.go:104\ngithub.com/pingcap/tidb/store/tikv.(*tikvSnapshot).Get\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/store/tikv/snapshot.go:387\ngithub.com/pingcap/tidb/structure.(*TxStructure).Get\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/structure/string.go:36\ngithub.com/pingcap/tidb/structure.(*TxStructure).GetInt64\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/structure/string.go:45\ngithub.com/pingcap/tidb/meta.(*Meta).GetSchemaVersion\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/meta/meta.go:260\ngithub.com/pingcap/tidb/domain.(*Domain).loadInfoSchema\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/domain/domain.go:102\ngithub.com/pingcap/tidb/domain.(*Domain).GetSnapshotInfoSchema\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/domain/domain.go:316\ngithub.com/pingcap/tidb/executor.(*SetExecutor).loadSnapshotInfoSchemaIfNeeded\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/set.go:314\ngithub.com/pingcap/tidb/executor.(*SetExecutor).setSysVariable\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/set.go:195\ngithub.com/pingcap/tidb/executor.(*SetExecutor).Next\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/set.go:108\ngithub.com/pingcap/tidb/executor.Next\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/executor.go:277\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:537\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:418\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:368\ngithub.com/pingcap/tidb/session.runStmt\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/session/session.go:1531\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/session/session.go:1426\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/driver_tidb.go:218\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/conn.go:1630\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/conn.go:1503\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/conn.go:1037\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/conn.go:795\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/server.go:477\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]

What is changed and how it works?

as title

Check List

Tests

• Manual test (TODO)

Side effects

Related changes

• Need to cherry-pick to the release branch

Release note

What problem does this PR solve?

To solve the #274.

What is changed and how it works?

STEP1:

Add the dump phase network speed limit feature.
Mainly modify the `writer` related code.

STEP2:

Modify the config code of DM later.

STEP3:

Update the document.

This PR is STEP1.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

Observing whether the network traffic in Grafana is stable.

Release note

No release note

What problem does this PR solve?

cherry-pick pingcap/tidb#29500 to release-5.2

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Possible performance regression
• Increased code complexity
• Breaking backward compatibility

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release note

What problem does this PR solve?

close #322

What is changed and how it works?

when meet a specific buggy version of mysql [8.0.3 8.0.23), use SQL show full tables to get all table names from upstream database.

As the average data size per row of data is not taken from the Show full table results, dumpling will set this value to 200000. This has the potential to cause the user to export too many or too few files, but I find this more acceptable than getting stuck in the dump stage.

Check List

Tests

• Unit test
• Integration test

Side effects

• Possible performance regression
• Increased code complexity

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release note

• fix pending on show table status in some mysql version

aurora does not support ftwrl when export data from aurora using dumlping --consistency =none , it will start a big transaction, and this will affect aurora performance.

Hope that when export data from aurora using dumlping --consistency =none, it will not start transactions, and affect aurora performance.

What did you do?

dumpling part table on tidb-cloud(dbaas). gc_life_time not support auto set when dumpling can't ping pd-server

What did you expect to see?

gc_life_time can be set adaptively

What did you see instead?

As gc progresses, dumpling exports an error

Versions of the cluster

v5.4.0

What did you do?

dumpling database using following commmand: tiup dumpling --host -u -p -P -B -o

What did you expect to see?

database being exported

What did you see instead?

Following error message: Error: use tiup install <database> to install component <database> first: component not installed.

If we use --database instead of -B, dumpling commond is executed successfully.

Versions of the cluster

Release version: v5.1.2 Git commit hash: 0d8ccce9feb254494e17b16d56bce8a2fc3e2469 Git branch: heads/refs/tags/v5.1.2 Build timestamp: 2021-09-27 02:47:55Z Go version: go version go1.16.4 linux/amd64

What problem does this PR solve?

Close #370

What is changed and how it works?

Add error message when --consistency is lock and --snapshot parameter is provided.

Check List

Tests

• Unit test
• Manual test (add detailed scripts or steps below)

Release note

No release note

Mariadb supports Unix Socket Authentication Plugin - https://mariadb.com/kb/en/authentication-plugin-unix-socket/ With it it is possible to login without password, for example:

[root@localhost ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5672
Server version: 10.4.19-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

dumpling seems to not support it:

[root@localhost ~]# ./dumpling -P3306
Release version: 
Git commit hash: 
Git branch:      
Build timestamp: 2021-06-14 04:26:05Z
Go version:      go version go1.16.5 linux/amd64

[2021/06/21 15:55:51.754 +00:00] [INFO] [versions.go:55] ["Welcome to dumpling"] ["Release Version"=] ["Git Commit Hash"=] ["Git Branch"=] ["Build timestamp"="2021-06-14 04:26:05"] ["Go Version"="go version go1.16.5 linux/amd64"]

create dumper failed: sql: SELECT version(): Error 1045: Access denied for user 'root'@'127.0.0.1' (using password: NO)

[root@localhost ~]# ./dumpling -hlocalhost -P3306
Release version: 
Git commit hash: 
Git branch:      
Build timestamp: 2021-06-14 04:26:05Z
Go version:      go version go1.16.5 linux/amd64

[2021/06/21 15:56:05.093 +00:00] [INFO] [versions.go:55] ["Welcome to dumpling"] ["Release Version"=] ["Git Commit Hash"=] ["Git Branch"=] ["Build timestamp"="2021-06-14 04:26:05"] ["Go Version"="go version go1.16.5 linux/amd64"]

create dumper failed: sql: SELECT version(): Error 1045: Access denied for user 'root'@'::1' (using password: NO)

It would be nice to support it.