Signed-off-by: crazycs520 [email protected]

close https://github.com/pingcap/ng-monitoring/issues/35

▶ curl "http://192.168.1.3:12020/continuous_profiling/group_profiles?begin_time=1639725850&end_time=1644836910"
[{"ts":1639725960,"profile_duration_secs":10,"state":"running","component_num":{"tidb":1,"pd":3,"tikv":0,"tiflash":0}}]% 

This PR also need to change the tidb-dashboard.

Signed-off-by: Zhenchi [email protected]

What problem does this PR solve?

Issue Number: close #136

What is changed and how it works?

1. Add function UpdateGlobalConfig
2. Use mutex to protect global config

Signed-off-by: Zhenchi [email protected]

What problem does this PR solve?

In master, if we want to query instances between ts 400 and ts 500, we query vm via /api/v1/query_range with arguments

query = last_over_time(instance[100s])
start = 400
end = 500
step = 100s

However, it will fetch instances from (300, 500].

What is changed and how it works?

In this patch, we query vm via /api/v1/query with arguments

query = last_over_time(instance[101s])
time = 500

Finally, It will fetch instances from [400, 500].

Besides, fetchSumFromTSDB can also change to the same way.

Signed-off-by: Zhenchi [email protected]

What problem does this PR solve?

Issue Number: close #71

What is changed and how it works?

Only on calling Recv, grpc client emits a grpc, then throws rpc error: code = Unimplemented.

Previously, I expected catching Unimplemented on calling Subscribe but that's not true.

This Patch

[2021/12/22 15:38:30.713 +08:00] [INFO] [manager.go:72] ["Top SQL is enabled"]
[2021/12/22 15:38:30.713 +08:00] [INFO] [scraper.go:60] ["starting to scrape top SQL from the component"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"]
[2021/12/22 15:38:30.713 +08:00] [INFO] [scraper.go:60] ["starting to scrape top SQL from the component"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"]
[2021/12/22 15:38:30.715 +08:00] [WARN] [scraper.go:296] ["failed to call Subscribe"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"] [error="rpc error: code = Unimplemented desc = "]
[2021/12/22 15:38:30.716 +08:00] [WARN] [scraper.go:276] ["failed to call Subscribe"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"] [error="rpc error: code = Unimplemented desc = unknown service tipb.TopSQLPubSub"]
[2021/12/22 15:38:34.716 +08:00] [WARN] [scraper.go:252] ["retry to scrape component"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"] [retried=1]
[2021/12/22 15:38:34.716 +08:00] [WARN] [scraper.go:252] ["retry to scrape component"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"] [retried=1]
[2021/12/22 15:38:34.717 +08:00] [WARN] [scraper.go:296] ["failed to call Subscribe"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"] [error="rpc error: code = Unimplemented desc = "]
[2021/12/22 15:38:34.717 +08:00] [WARN] [scraper.go:276] ["failed to call Subscribe"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"] [error="rpc error: code = Unimplemented desc = unknown service tipb.TopSQLPubSub"]
[2021/12/22 15:38:42.718 +08:00] [WARN] [scraper.go:252] ["retry to scrape component"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"] [retried=2]
[2021/12/22 15:38:42.718 +08:00] [WARN] [scraper.go:252] ["retry to scrape component"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"] [retried=2]
[2021/12/22 15:38:42.719 +08:00] [WARN] [scraper.go:296] ["failed to call Subscribe"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"] [error="rpc error: code = Unimplemented desc = "]
[2021/12/22 15:38:42.719 +08:00] [WARN] [scraper.go:276] ["failed to call Subscribe"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"] [error="rpc error: code = Unimplemented desc = unknown service tipb.TopSQLPubSub"]
[2021/12/22 15:38:48.634 +08:00] [INFO] [manager.go:118] ["update profile target info finished"] [update-count=0]
[2021/12/22 15:38:58.720 +08:00] [WARN] [scraper.go:252] ["retry to scrape component"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"] [retried=3]
[2021/12/22 15:38:58.720 +08:00] [WARN] [scraper.go:252] ["retry to scrape component"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"] [retried=3]
[2021/12/22 15:38:58.721 +08:00] [WARN] [scraper.go:296] ["failed to call Subscribe"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"] [error="rpc error: code = Unimplemented desc = "]
[2021/12/22 15:38:58.722 +08:00] [WARN] [scraper.go:276] ["failed to call Subscribe"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"] [error="rpc error: code = Unimplemented desc = unknown service tipb.TopSQLPubSub"]
[2021/12/22 15:39:30.722 +08:00] [WARN] [scraper.go:252] ["retry to scrape component"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"] [retried=4]
[2021/12/22 15:39:30.722 +08:00] [WARN] [scraper.go:252] ["retry to scrape component"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"] [retried=4]
[2021/12/22 15:39:30.723 +08:00] [WARN] [scraper.go:296] ["failed to call Subscribe"] [component="{\"name\":\"tikv\",\"ip\":\"127.0.0.1\",\"port\":20160,\"status_port\":20180}"] [error="rpc error: code = Unimplemented desc = "]
[2021/12/22 15:39:30.723 +08:00] [WARN] [scraper.go:276] ["failed to call Subscribe"] [component="{\"name\":\"tidb\",\"ip\":\"127.0.0.1\",\"port\":4000,\"status_port\":10080}"] [error="rpc error: code = Unimplemented desc = unknown service tipb.TopSQLPubSub"]

What problem does this PR solve?

Issue Number: close #xxx

What is changed and how it works?

cherry-pick https://github.com/pingcap/ng-monitoring/pull/115

https://github.com/pingcap/ng-monitoring/pull/119

Signed-off-by: crazycs520 [email protected]

What problem does this PR solve?

Issue Number: close #57 Issue Number: close #56 Issue Number: close #90

What is changed and how it works?

Signed-off-by: mornyx [email protected]

What is changed and how it works?

• Add two metrics: sql_exec_count and sql_duration.
• Implement write and query for new metrics.
• Enhance integration tests of TopSQL.

Related PR

• https://github.com/pingcap/tipb/pull/250

This is an automated cherry-pick of #137

Signed-off-by: Zhenchi [email protected]

What problem does this PR solve?

Issue Number: close #136

What is changed and how it works?

1. Add function UpdateGlobalConfig
2. Use mutex to protect global config

Signed-off-by: crazycs520 [email protected]

What problem does this PR solve?

close: #120

Related issue: https://github.com/genjidb/genji/issues/454

https://discuss.dgraph.io/t/badgerdb-consume-too-much-disk-space/17070

Related repo: https://github.com/crazycs520/genji/tree/v0.13.0.bugfix

What is changed and how it works?

The bug was caused by genjidb, but genjidb doesn't support badger now, so I have to fork the genjidb then fix in my repo, then replace it in go.mod

Signed-off-by: crazycs520 [email protected]

What problem does this PR solve?

--- FAIL: TestProfileStorage (0.54s)
    store_test.go:111:
                Error Trace:    store_test.go:111
                                                        store_test.go:29
                Error:          Not equal:
                                expected: 1644219544
                                actual  : 1644219543
                Test:           TestProfileStorage

What is changed and how it works?

Signed-off-by: Zhenchi [email protected]

What problem does this PR solve?

ngm occupies much memory when it collects lots of conprof data. It is mainly caused by retention of conprof. When executing DELETE FROM conprof_table WHERE ts <= ?, genji does an index scan through iterator, which acts prefetch due to the default option. The prefetch loads conprof data to memory through mmap but never read them. We benefit little from prefetch, but it does cause memory usage to increase.

What is changed and how it works?

• Disable prefetch.
• Tweak options of badger.

We hope to bring Continuous Profiling to TiCDC, this will be divided into two steps:

• [x] Add TiCDC to the ng-monitoring backend. Although not yet shown on the web frontend, we can simply click the "Download" button to get all the Profiles (including TiCDC) at a certain point in time.

• [ ] Let the web frontend of TiDB Dashboard display TiCDC related profiles.

See: https://github.com/pingcap/tiflash/issues/5687

Continuous Profiling has been enabled by default since v6.1.0, so TiDB/TiKV/TiFlash will continue to trigger CPU profiling by default since v6.1.0. The CPU profiler of TiKV/TiFlash is pprof-rs. pprof-rs registers a signal handler during CPU profiling and periodically triggers the SIGPROF signal. After the SIGPROF signal is triggered, the signal handler is dispatched to the business thread for execution to sample the call stack of the current thread. Signal handler also makes system calls through glibc during execution, so it has a chance to affect errno. After the signal handler is executed, the business thread may get the wrong errno. However, errno has been protected in the current version of pprof-rs, so there is still no key evidence that the errno was modified by pprof-rs. The current judgment is based on two facts:

1. The problem only occurs while profiling is on
2. pprof-rs has a very similar issue

So, Continuous Profiling should be disabled by default until tiflash can be safely profiled.

Bug Report

Please answer these questions before submitting your issue. Thanks!

What did you do?

Open topsql feature from March 21.

What did you expect to see?

topsql's retention time is 1 month and it does work, it should delete data that exceed retention time.

What did you see instead?

data for march 21 is still available.

What version of TiDB Dashboard are you using (./tidb-dashboard --version)?

6.0

This issue is from https://github.com/pingcap/tidb-dashboard/issues/1218

The storage engine is evolved as the follows:

Use a KV that supports ZSTD to achieve max compression → Use genjidb for easier access over that KV engine

However, as we are now actually not using ZSTD for block-compressing, but compressing at a per-profile level, the genjidb + badger is not the only choice any more. For example, as the most widely deployed database engine, Sqlite may be a better choice.

I'm using Ctrl+C to gracefully quit the ng-monitoring and starting a new one:

[GIN] 2022/02/08 - 14:43:48 | 404 |         792ns | 192.168.126.218 | GET      "/"
^C[2022/02/08 14:46:06.297 +08:00] [INFO] [main.go:108] ["received signal"] [sig=interrupt]
[2022/02/08 14:46:06.297 +08:00] [INFO] [http.go:79] ["shutting down http server"]
[2022/02/08 14:46:06.297 +08:00] [INFO] [http.go:81] ["http server is down"]
[2022/02/08 14:46:06.298 +08:00] [INFO] [default_subscriber.go:48] ["stopping scrapers"]
[2022/02/08 14:46:06.298 +08:00] [INFO] [default_subscriber.go:51] ["stop scrapers successfully"]
[2022/02/08 14:46:06.298 +08:00] [INFO] [database.go:20] ["Stopping timeseries database"]
[2022/02/08 14:46:06.313 +08:00] [INFO] [database.go:22] ["Stop timeseries database successfully"]
[2022/02/08 14:46:06.313 +08:00] [INFO] [database.go:24] ["Stopping document database"]
[2022/02/08 14:46:06.313 +08:00] [INFO] [document.go:51] ["badger stop running value log gc loop"]
[2022/02/08 14:46:06.384 +08:00] [INFO] [database.go:26] ["Stop document database successfully"]

However the topology is not cleaned up in time, so that TiDB Dashboard keeps connect to the non-existing ng-monitoring server:

$ etcdctl get /topology --prefix
/topology/ng-monitoring/192.168.126.218:12020/info   <-- new
{"git_hash":"1afcaa990af5c65b222e0ab59171867248645f4a","ip":"192.168.126.218","listening_port":12020,"start_timestamp":1644302767}
/topology/ng-monitoring/192.168.126.218:12020/ttl   <-- new
1644302767649395000
/topology/ng-monitoring/192.168.3.105:12020/info   <-- old, gracefully exited
{"git_hash":"1afcaa990af5c65b222e0ab59171867248645f4a","ip":"192.168.3.105","listening_port":12020,"start_timestamp":1644225105}
/topology/ng-monitoring/192.168.3.105:12020/ttl   <-- old, gracefully exited
1644302745923627000
/topology/tidb/127.0.0.1:4000/info
{"version":"v5.3.0","git_hash":"4a1b2e9fe5b5afb1068c56de47adb07098d768d6","ip":"127.0.0.1","status_port":10080,"deploy_path":"/Users/breezewish/.tiup/components/tidb/v5.3.0","start_timestamp":1644224846,"labels":{}}
/topology/tidb/127.0.0.1:4000/ttl
1644302786725919000

This will cause problems when user scales-in and scales-out (switch) the ngm node.