Hey there,
Not sure if you're open to contributions nor if this is the right way to do it...
But while trying out the project with a simple video (this one to be exact), I faced the following error when running promtool tsdb create-blocks-from openmetrics [...]
:
prometheus_1 | level=error ts=2021-08-12T03:12:17.882Z caller=db.go:766 component=tsdb msg=reloadBlocks err="invalid block sequence: block time ranges overlap: [mint: 1628676000000, maxt: 1628676299001, range: 4m59s, blocks: 2]: <ulid: 01FCW4B0D0C46EF2YZDG3YWJ0J, mint: 1628676000000, maxt: 1628676299001, range: 4m59s>, <ulid: 01FCW52YXKKZ8HPXZY2QY5R5FS, mint: 1628676000000, maxt: 1628676299001, range: 4m59s>\n[mint: 1628640000000, maxt: 1628647199001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW521C9JHXC2S7MMEWN72AQ, mint: 1628640000000, maxt: 1628647199001, range: 1h59m59s>, <ulid: 01FCW514MC9DAVTAJBBSTDR5P7, mint: 1628640000000, maxt: 1628661599001, range: 5h59m59s>\n[mint: 1628647200000, maxt: 1628654399001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW514MC9DAVTAJBBSTDR5P7, mint: 1628640000000, maxt: 1628661599001, range: 5h59m59s>, <ulid: 01FCW527GHM1V3BTMGWJXSZ3JC, mint: 1628647200000, maxt: 1628654399001, range: 1h59m59s>\n[mint: 1628654400000, maxt: 1628661599001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW514MC9DAVTAJBBSTDR5P7, mint: 1628640000000, maxt: 1628661599001, range: 5h59m59s>, <ulid: 01FCW52DHMMZCKX6PPTAEVVM9A, mint: 1628654400000, maxt: 1628661599001, range: 1h59m59s>\n[mint: 1628661600000, maxt: 1628668799001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW52KK7F98KR9N1G9D4VPYG, mint: 1628661600000, maxt: 1628668799001, range: 1h59m59s>, <ulid: 01FCW5153RF2BMBBGXMCAW5916, mint: 1628661600000, maxt: 1628668799001, range: 1h59m59s>\n[mint: 1628668800000, maxt: 1628675999001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW52ST0Q92GDK8DC85CYFNS, mint: 1628668800000, maxt: 1628675999001, range: 1h59m59s>, <ulid: 01FCW4AV94BCM4ZD6RHK1Z3M37, mint: 1628668800000, maxt: 1628675999001, range: 1h59m59s>"
prometheus_1 | level=info ts=2021-08-12T03:12:50.308Z caller=compact.go:692 component=tsdb msg="Found overlapping blocks during compaction" ulid=01FCW5J83D5PWCKMJ5P9FD4H3X
prometheus_1 | level=info ts=2021-08-12T03:12:50.454Z caller=compact.go:454 component=tsdb msg="compact blocks" count=4 mint=1628640000000 maxt=1628661599001 ulid=01FCW5J83D5PWCKMJ5P9FD4H3X sources="[01FCW514MC9DAVTAJBBSTDR5P7 01FCW521C9JHXC2S7MMEWN72AQ 01FCW527GHM1V3BTMGWJXSZ3JC 01FCW52DHMMZCKX6PPTAEVVM9A]" duration=245.7086ms
prometheus_1 | level=error ts=2021-08-12T03:12:52.058Z caller=db.go:780 component=tsdb msg="compaction failed" err="reloadBlocks blocks: invalid block sequence: block time ranges overlap: [mint: 1628661600000, maxt: 1628668799001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW5153RF2BMBBGXMCAW5916, mint: 1628661600000, maxt: 1628668799001, range: 1h59m59s>, <ulid: 01FCW52KK7F98KR9N1G9D4VPYG, mint: 1628661600000, maxt: 1628668799001, range: 1h59m59s>\n[mint: 1628668800000, maxt: 1628675999001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW4AV94BCM4ZD6RHK1Z3M37, mint: 1628668800000, maxt: 1628675999001, range: 1h59m59s>, <ulid: 01FCW52ST0Q92GDK8DC85CYFNS, mint: 1628668800000, maxt: 1628675999001, range: 1h59m59s>\n[mint: 1628676000000, maxt: 1628676299001, range: 4m59s, blocks: 2]: <ulid: 01FCW52YXKKZ8HPXZY2QY5R5FS, mint: 1628676000000, maxt: 1628676299001, range: 4m59s>, <ulid: 01FCW4B0D0C46EF2YZDG3YWJ0J, mint: 1628676000000, maxt: 1628676299001, range: 4m59s>"
prometheus_1 | level=error ts=2021-08-12T03:14:53.812Z caller=db.go:766 component=tsdb msg=reloadBlocks err="invalid block sequence: block time ranges overlap: [mint: 1628676000000, maxt: 1628676299001, range: 4m59s, blocks: 2]: <ulid: 01FCW4B0D0C46EF2YZDG3YWJ0J, mint: 1628676000000, maxt: 1628676299001, range: 4m59s>, <ulid: 01FCW52YXKKZ8HPXZY2QY5R5FS, mint: 1628676000000, maxt: 1628676299001, range: 4m59s>\n[mint: 1628640000000, maxt: 1628647199001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW521C9JHXC2S7MMEWN72AQ, mint: 1628640000000, maxt: 1628647199001, range: 1h59m59s>, <ulid: 01FCW514MC9DAVTAJBBSTDR5P7, mint: 1628640000000, maxt: 1628661599001, range: 5h59m59s>\n[mint: 1628647200000, maxt: 1628654399001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW514MC9DAVTAJBBSTDR5P7, mint: 1628640000000, maxt: 1628661599001, range: 5h59m59s>, <ulid: 01FCW527GHM1V3BTMGWJXSZ3JC, mint: 1628647200000, maxt: 1628654399001, range: 1h59m59s>\n[mint: 1628654400000, maxt: 1628661599001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW514MC9DAVTAJBBSTDR5P7, mint: 1628640000000, maxt: 1628661599001, range: 5h59m59s>, <ulid: 01FCW52DHMMZCKX6PPTAEVVM9A, mint: 1628654400000, maxt: 1628661599001, range: 1h59m59s>\n[mint: 1628661600000, maxt: 1628668799001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW52KK7F98KR9N1G9D4VPYG, mint: 1628661600000, maxt: 1628668799001, range: 1h59m59s>, <ulid: 01FCW5153RF2BMBBGXMCAW5916, mint: 1628661600000, maxt: 1628668799001, range: 1h59m59s>\n[mint: 1628668800000, maxt: 1628675999001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW4AV94BCM4ZD6RHK1Z3M37, mint: 1628668800000, maxt: 1628675999001, range: 1h59m59s>, <ulid: 01FCW52ST0Q92GDK8DC85CYFNS, mint: 1628668800000, maxt: 1628675999001, range: 1h59m59s>"
[...]
It went away simply enabling the storage.tsdb.allow-overlapping-blocks
as described in the docs: https://prometheus.io/docs/prometheus/latest/storage/#usage
A simple warning is shown now and backfill proceeds as expected
prometheus_1 | level=info ts=2021-08-12T03:36:56.469Z caller=compact.go:692 component=tsdb msg="Found overlapping blocks during compaction" ulid=01FCW6YCC5TREHFSPCD8F6KD6J
prometheus_1 | level=info ts=2021-08-12T03:36:57.462Z caller=compact.go:454 component=tsdb msg="compact blocks" count=8 mint=1629266400000 maxt=1629331199001 ulid=01FCW6YCC5TREHFSPCD8F6KD6J sources="[01FCW6WGGW3GGZ1F6X9KYBBDKK 01FCW6WQ6K2BFBKDDKR45QAAAQ 01FCW6WY7CPTAJFEVT6NBRCZK9 01FCW6X5393PBK8C3CEFX1WRDX 01FCW6XC0MR7CKVTMTZK2B1P12 01FCW6XK4MTGEY093JD5A78Q85 01FCW6XRPE4C8C53MPNB5J50M0 01FCW6Y5NZZRMHFQ1TXHR88JVT]" duration=1.0261382s
prometheus_1 | level=warn ts=2021-08-12T03:36:57.597Z caller=db.go:1084 component=tsdb msg="Overlapping blocks found during reloadBlocks" detail="[mint: 1629331200000, maxt: 1629338399001, range: 1h59m59s, blocks: 2]: <ulid: 01FCW6YCDRTMSPCZQWM05FXQ2J, mint: 1629331200000, maxt: 1629338399001, range: 1h59m59s>, <ulid: 01FCW64189DDKV7ACDJG5V0RV9, mint: 1629331200000, maxt: 1629352799001, range: 5h59m59s>"
prometheus_1 | level=info ts=2021-08-12T03:36:57.617Z caller=db.go:1239 component=tsdb msg="Deleting obsolete block" block=01FCW6WGGW3GGZ1F6X9KYBBDKK
prometheus_1 | level=info ts=2021-08-12T03:36:57.638Z caller=db.go:1239 component=tsdb msg="Deleting obsolete block" block=01FCW6WY7CPTAJFEVT6NBRCZK9
[...]
Let me know if this PR needs any modifications.