Implementation of the Filecoin protocol, written in Go

Last update: Jan 9, 2023

Comments: 16

Project Lotus - 莲

Lotus is an implementation of the Filecoin Distributed Storage Network. For more details about Filecoin, check out the Filecoin Spec.

Building & Documentation

For instructions on how to build, install and setup lotus, please visit https://docs.filecoin.io/get-started/lotus.

Reporting a Vulnerability

Please send an email to [email protected]. See our security policy for more details.

Related packages

These repos are independent and reusable modules, but are tightly integrated into Lotus to make up a fully featured Filecoin implementation:

go-fil-markets which has its own kanban work tracker available here
specs-actors which has its own kanban work tracker available here

Contribute

Lotus is a universally open project and welcomes contributions of all kinds: code, docs, and more. However, before making a contribution, we ask you to heed these recommendations:

If the proposal entails a protocol change, please first submit a Filecoin Improvement Proposal.
If the change is complex and requires prior discussion, open an issue or a discussion to request feedback before you start working on a pull request. This is to avoid disappointment and sunk costs, in case the change is not actually needed or accepted.
Please refrain from submitting PRs to adapt existing code to subjective preferences. The changeset should contain functional or technical improvements/enhancements, bug fixes, new features, or some other clear material contribution. Simple stylistic changes are likely to be rejected in order to reduce code churn.

When implementing a change:

Adhere to the standard Go formatting guidelines, e.g. Effective Go. Run go fmt.
Stick to the idioms and patterns used in the codebase. Familiar-looking code has a higher chance of being accepted than eerie code. Pay attention to commonly used variable and parameter names, avoidance of naked returns, error handling patterns, etc.
Comments: follow the advice on the Commentary section of Effective Go.
Minimize code churn. Modify only what is strictly necessary. Well-encapsulated changesets will get a quicker response from maintainers.
Lint your code with golangci-lint (CI will reject your PR if unlinted).
Add tests.
Title the PR in a meaningful way and describe the rationale and the thought process in the PR description.
Write clean, thoughtful, and detailed commit messages. This is even more important than the PR description, because commit messages are stored inside the Git history. One good rule is: if you are happy posting the commit message as the PR description, then it's a good commit message.

License

Dual-licensed under MIT + Apache 2.0

Owner

Filecoin

https://github.com/filecoin-project/lotus https://docs.filecoin.io/get-started/lotus/

Comments

Arm64 Raspberry Pi Build Failure
Describe the bug When trying to build the lotus project from source files I run into this error:

error[E0433]: failed to resolve: use of undeclared type or module `cc` --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/fil-sapling-crypto-0.6.0/build.rs:6:9 | 6 | cc::Build::new() | ^^ use of undeclared type or module `cc` error: aborting due to previous error

To Reproduce Steps to reproduce the behavior:

followed the steps in this issue: https://github.com/filecoin-project/lotus/issues/1779

export RUSTFLAGS="-C target-cpu=native -g" export FFI_BUILD_FROM_SOURCE=1 make clean deps bench

Expected behavior The lotus project to build without errors

Screenshots Someone already reported the issue on an upstream repo. https://github.com/zcash-hackworks/sapling-crypto/issues/104

Version (run lotus --version): unable to compile the latest version

Additional context Add any other context about the problem here. Arm64 Architecture
ChainGetTipSetByHeight method synchronization lacks message

The ChainGetTipSetByHeight method cannot synchronize the messages under all miners and there are references bug i used the ChainGetTipSetByHeight rpc Can't sync to that transaction

the method is follow img:
updating to new datastore/blockstore code with contexts
Status

The following deps need to be tagged, as we have go mod versions pointing at unstable things:

github.com/drand/drand

The following deps also exist in the lotus-soup go.mod:

github.com/drand/drand

Mined blocks became orphan blocks

Describe the bug

We mined some blocks，but don't know why these blocks are not exists on chain.

2020-12-06T01:23:36.133Z        INFO    storageminer    storage/miner.go:251    Computing WinningPoSt ;[{SealProof:8 SectorNumber:25272 SealedCID:bagboea4b5abcbe2cyns3zmw3mhhlftwx3im76ytulqdpfkivvmh7tf5bub46qzrk}]; [194 38 156 168 46 37 44 32 46 145 33 78 103 176 212 16 46 155 183 43 24 72 86 19 54 56 37 183 139 26 184 11]
2020-12-06T01:23:41.759Z        INFO    storageminer    storage/miner.go:258    GenerateWinningPoSt took 5.626304116s
2020-12-06T01:23:42.157Z        INFO    miner   miner/miner.go:448      mined new block {"cid": "bafy2bzaceaj6mkgqeh6qrnl4jrqwxptuanijcpgg6eo2dzspnxewj3bzbhp7k", "height": "297048", "miner": "f03339", "parents": ["f07830","f02520","f030347","f02777","f020331","f020522"], "took": 6.035145888}

2020-12-05T23:20:06.100Z        INFO    storageminer    storage/miner.go:251    Computing WinningPoSt ;[{SealProof:3 SectorNumber:16435 SealedCID:bagboea4b5abcb4t4fybuumey4xfiefynlhqqrdlmzfoyg7uqwym7vrx75d6uwzc5}]; [182 6 130 38 137 107 109 218 22 88 224 234 22 86 97 212 80 201 160 117 242 136 51 129 201 109 113 41 181 201 28 175]
2020-12-05T23:20:07.577Z        WARN    sectors storage-sealing/states_sealing.go:355   revert in interactive commit sector step
2020-12-05T23:20:11.758Z        INFO    storageminer    storage/miner.go:258    GenerateWinningPoSt took 5.657667875s
2020-12-05T23:20:12.203Z        INFO    miner   miner/miner.go:448      mined new block {"cid": "bafy2bzaceceuzscc5kxhkc4eytak4wlbmemdd4m3e7dqlmmsrb37hva4p7zri", "height": "296801", "miner": "f03339", "parents": ["f034548","f023530","f022374","f010253"], "took": 6.113739295}

1.18.0, the miner process crashes during execution of the AddPiece phase

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
[X] This is not a new feature request. If it is, please file a feature request instead.
[X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[X] I did not make any code changes to lotus.

Lotus component

[ ] lotus daemon - chain sync
[ ] lotus miner - mining and block production
[X] lotus miner/worker - sealing
[ ] lotus miner - proving(WindowPoSt)
[X] lotus miner/market - storage deal
[ ] lotus miner/market - retrieval deal
[ ] lotus miner/market - data transfer
[ ] lotus client
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[ ] Other

Lotus Version

lotus --version
lotus version 1.18.0+mainnet+git.bd10bdf99
lotus-miner --version
lotus-miner version 1.18.0+mainnet+git.bd10bdf99

Describe the Bug

From time to time, the miner process crashes during (or when trying to) execute the AddPiece phase. In the time that has passed since the network upgrade, this has already happened twice.

Logging Information

Last lines of miner.log:

2022-12-02T03:26:40.285+0300    INFO    miner   miner/miner.go:475      completed mineOne       {"tookMilliseconds": 6, "forRound": 2387814, "baseEpoch": 2387813, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2483659, "lookbackEpochs": 900, "networkPowerAtLookback": "21373846934587277312", "minerPowerAtLookback": "3248215606362112", "isEligible": true, "isWinner": false, "error": null}
2022-12-02T03:27:10.272+0300    INFO    miner   miner/miner.go:475      completed mineOne       {"tookMilliseconds": 8, "forRound": 2387815, "baseEpoch": 2387814, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2483660, "lookbackEpochs": 900, "networkPowerAtLookback": "21374157681872076800", "minerPowerAtLookback": "3248215606362112", "isEligible": true, "isWinner": false, "error": null}
2022-12-02T03:27:30.315+0300    INFO    sectors pipeline/input.go:286   Adding piece for deal 17660229 (publish msg: bafy2bzacedj5ceycbivgvvx3k22lpntnjnmyiirowgft6izgr2amarwyrdnys)
2022-12-02T03:27:30.316+0300    INFO    sectors pipeline/input.go:286   Adding piece for deal 17660230 (publish msg: bafy2bzacedj5ceycbivgvvx3k22lpntnjnmyiirowgft6izgr2amarwyrdnys)


Last lines of miner_rust.log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1ab85be]

goroutine 269380609 [running]:
github.com/filecoin-project/lotus/storage/pipeline.(*Sealing).updateInput.func1(0x3a28500?)
        /home/admfc/lotus/storage/pipeline/input.go:449 +0xbe
github.com/filecoin-project/lotus/storage/pipeline.(*Sealing).updateInput(0xc00055c3c0, {0x46836a8, 0xc04558b710}, 0x8?)
        /home/admfc/lotus/storage/pipeline/input.go:491 +0x1035
github.com/filecoin-project/lotus/storage/pipeline.(*Sealing).addPendingPiece.func2()
        /home/admfc/lotus/storage/pipeline/input.go:401 +0x7b
created by github.com/filecoin-project/lotus/storage/pipeline.(*Sealing).addPendingPiece
        /home/admfc/lotus/storage/pipeline/input.go:399 +0x267

Repo Steps

Not sure if the problem is reproducible. There are two servers on the same site that are almost identical in hardware stuffing, both on the same software version. The 32GiB miner with the Threadripper PRO 3945WX processor is experiencing a problem, the 64GiB miner with the 3955WX processor has not experienced it yet. But in general -

Install release 1.18.0
Accept deals.
Observe the problem.

GPU stuck at P2

Describe the bug

This problem is an occasional problem, When the worker is started for the first time, there will be a high probability of problems, Then the probability of encountering this problem after restarting will decrease

Our GPU: Geforce RTX 2080 Ti

worker log:

2020-11-24T01:18:01.561 INFO filcrypto::proofs::api > seal_pre_commit_phase2: start
2020-11-24T01:18:01.575 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:start
2020-11-24T01:18:01.603 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:finish
2020-11-24T01:18:01.637 INFO filecoin_proofs::api::seal > seal_pre_commit_phase2:start
2020-11-24T01:18:01.708 INFO storage_proofs_porep::stacked::vanilla::proof > replicate_phase2
2020-11-24T01:18:01.708 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree c using the GPU
2020-11-24T01:18:01.708 INFO storage_proofs_porep::stacked::vanilla::proof > Building column hashes
2020-11-24T01:18:01.736 INFO neptune::cl > getting context for ~Index(0)
2020-11-24T01:18:01.793 WARN neptune::cl > Cannot get device list for platform: Clover!
2020-11-24T01:18:01.793 WARN neptune::cl > Cannot get device list for platform: Clover!
2020-11-24T01:19:55.835 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 1/8 of length 153391689
2020-11-24T01:21:31.248 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 2/8 of length 153391689
2020-11-24T01:23:04.417 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 3/8 of length 153391689
2020-11-24T01:24:38.587 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 4/8 of length 153391689
2020-11-24T01:26:13.404 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 5/8 of length 153391689
2020-11-24T01:27:46.782 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 6/8 of length 153391689
2020-11-24T01:29:27.555 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 7/8 of length 153391689
2020-11-24T01:31:13.127 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 8/8 of length 153391689
2020-11-24T01:31:18.598 INFO storage_proofs_porep::stacked::vanilla::proof > tree_c done
2020-11-24T01:31:18.598 INFO storage_proofs_porep::stacked::vanilla::proof > building tree_r_last
2020-11-24T01:31:18.598 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree r last using the GPU
2020-11-24T01:31:19.224 INFO neptune::cl > getting context for ~Index(0)
2020-11-24T01:31:19.369 WARN neptune::cl > Cannot get device list for platform: Clover!
2020-11-24T10:43:56.641+0800    ^[[33mWARN^[[0m main    lotus-seal-worker/main.go:421   Shutting down...
2020-11-24T10:43:56.655+0800    ^[[33mWARN^[[0m main    lotus-seal-worker/main.go:98    http: Server closed

dmesg log:

[Tue Nov 24 01:18:26 2020] NVRM: GPU at PCI:0000:81:00: GPU-77feb6df-eb6f-ae6d-8f89-5f84fb7c3e40
[Tue Nov 24 01:18:26 2020] NVRM: GPU Board Serial Number:
[Tue Nov 24 01:18:26 2020] NVRM: Xid (PCI:0000:81:00): 13, pid=30010, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 0): Out Of Range Address
[Tue Nov 24 01:18:26 2020] NVRM: Xid (PCI:0000:81:00): 13, pid=30010, Graphics Exception: ESR 0x514730=0xc01000e 0x514734=0x20 0x514728=0x4c1eb72 0x51472c=0x174
[Tue Nov 24 01:18:26 2020] NVRM: Xid (PCI:0000:81:00): 43, pid=30398, Ch 00000008

I compared the logs of the worker after the first startup and restart, I found one place is different：

first startup:

2020-11-24T01:18:01.561 INFO filcrypto::proofs::api > seal_pre_commit_phase2: start
2020-11-24T01:18:01.575 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:start
2020-11-24T01:18:01.603 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:finish
2020-11-24T01:18:01.637 INFO filecoin_proofs::api::seal > seal_pre_commit_phase2:start
2020-11-24T01:18:01.708 INFO storage_proofs_porep::stacked::vanilla::proof > replicate_phase2
2020-11-24T01:18:01.708 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree c using the GPU
2020-11-24T01:18:01.708 INFO storage_proofs_porep::stacked::vanilla::proof > Building column hashes
2020-11-24T01:18:01.736 INFO neptune::cl > getting context for ~Index(0)

----------------------------------------------------------------------------------------------------------------

2020-11-24T01:18:01.793 WARN neptune::cl > Cannot get device list for platform: Clover!
2020-11-24T01:18:01.793 WARN neptune::cl > Cannot get device list for platform: Clover!

## Appeared here twice
----------------------------------------------------------------------------------------------------------------

2020-11-24T01:19:55.835 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 1/8 of length 153391689
2020-11-24T01:21:31.248 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 2/8 of length 153391689
2020-11-24T01:23:04.417 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 3/8 of length 153391689
2020-11-24T01:24:38.587 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 4/8 of length 153391689
2020-11-24T01:26:13.404 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 5/8 of length 153391689
2020-11-24T01:27:46.782 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 6/8 of length 153391689
2020-11-24T01:29:27.555 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 7/8 of length 153391689
2020-11-24T01:31:13.127 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 8/8 of length 153391689
2020-11-24T01:31:18.598 INFO storage_proofs_porep::stacked::vanilla::proof > tree_c done
2020-11-24T01:31:18.598 INFO storage_proofs_porep::stacked::vanilla::proof > building tree_r_last
2020-11-24T01:31:18.598 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree r last using the GPU
2020-11-24T01:31:19.224 INFO neptune::cl > getting context for ~Index(0)
2020-11-24T01:31:19.369 WARN neptune::cl > Cannot get device list for platform: Clover!

restart:

2020-11-24T15:42:33.077 INFO filecoin_proofs::api::seal > seal_pre_commit_phase2:start
2020-11-24T15:42:33.079 INFO storage_proofs_porep::stacked::vanilla::proof > replicate_phase2
2020-11-24T15:42:33.079 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree c using the GPU
2020-11-24T15:42:33.079 INFO storage_proofs_porep::stacked::vanilla::proof > Building column hashes
2020-11-24T15:42:33.079 INFO neptune::cl > getting context for ~Index(0)

----------------------------------------------------------------------------------------------------------------

2020-11-24T15:42:33.084 WARN neptune::cl > Cannot get device list for platform: Clover!

## Appeared here once
----------------------------------------------------------------------------------------------------------------

2020-11-24T15:44:44.044 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 1/8 of length 153391689
2020-11-24T15:47:00.733 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 2/8 of length 153391689
2020-11-24T15:49:14.234 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 3/8 of length 153391689
2020-11-24T15:51:29.922 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 4/8 of length 153391689
2020-11-24T15:53:44.894 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 5/8 of length 153391689
2020-11-24T15:55:59.994 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 6/8 of length 153391689
2020-11-24T15:58:15.629 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 7/8 of length 153391689
2020-11-24T16:00:31.432 INFO storage_proofs_porep::stacked::vanilla::proof > persisting base tree_c 8/8 of length 153391689
2020-11-24T16:00:36.050 INFO storage_proofs_porep::stacked::vanilla::proof > tree_c done
2020-11-24T16:00:36.050 INFO storage_proofs_porep::stacked::vanilla::proof > building tree_r_last
2020-11-24T16:00:36.050 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree r last using the GPU
2020-11-24T16:00:36.632 INFO neptune::cl > getting context for ~Index(0)
2020-11-24T16:00:36.632 WARN neptune::cl > Cannot get device list for platform: Clover!
2020-11-24T16:00:45.490 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_r_last with GPU 1/8
 ......... snip ..............

Lotus-bench results thread (v20 params)
This issue is a place to put lotus-bench results for v20 params.

To best help us, run four tests:

Start by installing build dependencies from https://docs.lotu.sh/en+getting-started

git clone https://github.com/filecoin-project/lotus.git cd lotus make build bench ./bench --sector-size=1073741824 ./bench --sector-size=1073741824 --no-gpu # Only run these with > 64GiB of ram, recommended 128G ./bench --sector-size=34359738368 ./bench --sector-size=34359738368 --no-gpu

Additionally, please tell us what CPU, GPU, and memory (including speed) you have in your setup.

Previous (v19) thread - https://github.com/filecoin-project/lotus/issues/694
Cannot perform any retrieval for 32GB files

Describe the bug I tried several retrievals to several different miners for the deals I made. None of them went through. I do not believe they all offline. I suspect it has something to do with the deal size. These are all 32GB offline deals, made with --fast-retrieval=false.

Example 1 - stuck after DealStatusOngoing Provider is doing unsealing but it takes hours without going further

lotus client retrieve --miner f024008 bafykbzaced6noeziyglm2frycaknydeijbimvhpk72eimhfupoxd72dehp2k4 abc

Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew) Recv: 0 B, Paid 0 FIL, ClientEventDealProposed (DealStatusWaitForAcceptance) Recv: 0 B, Paid 0 FIL, ClientEventDealAccepted (DealStatusAccepted) Recv: 0 B, Paid 0 FIL, ClientEventPaymentChannelCreateInitiated (DealStatusPaymentChannelCreating) Recv: 0 B, Paid 0 FIL, ClientEventPaymentChannelReady (DealStatusPaymentChannelAllocatingLane) Recv: 0 B, Paid 0 FIL, ClientEventLaneAllocated (DealStatusOngoing)

Example 2 - unmarshalling error The provider told me his miner crashed upon receiving my retrieval request

lotus client retrieve --miner f083550 bafykbzacecu4qt4tlr5vqojgtlt553rbskqremq67z37x5cufaonjqfh5y7no mysql-2016-04-19.tar.gz.partbl 2021-01-25T04:59:20.145Z WARN rpc [email protected]/client.go:541 unmarshaling failed {"message": "{"Err":"exhausted 5 attempts but failed to open stream, err: failed to dial 12D3KooWFJ6iPAiW82pR7REB8pJfKAsoKoEAhgWKYduoQT1734a9: all dials failed\n * [/ip4/127.0.0.1/tcp/45989] dial tcp4 127.0.0.1:45989: connect: connection refused\n * [/ip6/::1/tcp/41337] dial tcp6 [::1]:41337: connect: connection refused\n * [/ip4/59.12.56.212/tcp/45989] dial tcp4 59.12.56.212:45989: connect: connection refused\n * [/ip4/59.12.56.215/tcp/45989] dial tcp4 0.0.0.0:33463-\u003e59.12.56.215:45989: i/o timeout","Root":null,"Piece":null,"Size":0,"MinPrice":"\u003cnil\u003e","UnsealPrice":"\u003cnil\u003e","PaymentInterval":0,"PaymentIntervalIncrease":0,"Miner":"f083550","MinerPeer":{"Address":"f083550","ID":"12D3KooWFJ6iPAiW82pR7REB8pJfKAsoKoEAhgWKYduoQT1734a9","PieceCID":null}}"} ERROR: RPC client error: unmarshaling result: failed to parse big string: '"\u003cnil\u003e"'

Example 3 - incomplete response One of the provider told me his miner is set as allowUnseal=false but his worker is set as allowUnseal=true, yet the retrieval failed before the worker picked up the unseal.

lotus client retrieve --miner f064218 baga6ea4seaqgsbrsupr6az5zf2yjqai5t4xtgn7lf2zd62wnxxq2mgyhlvbxyda publicdomainmovies.tar.05

Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew) Recv: 0 B, Paid 0 FIL, ClientEventDealProposed (DealStatusWaitForAcceptance) Recv: 0 B, Paid 0 FIL, ClientEventUnsealPaymentRequested (DealStatusAccepted) Recv: 0 B, Paid 0 FIL, ClientEventPaymentChannelAddingFunds (DealStatusPaymentChannelAllocatingLane) Recv: 0 B, Paid 0 FIL, ClientEventLaneAllocated (DealStatusOngoing) Recv: 0 B, Paid 0 FIL, ClientEventPaymentRequested (DealStatusFundsNeeded) Recv: 0 B, Paid 0 FIL, ClientEventSendFunds (DealStatusSendFunds) Recv: 0 B, Paid 0.1 FIL, ClientEventPaymentSent (DealStatusOngoing) Recv: 0 B, Paid 0.1 FIL, ClientEventDataTransferError (DealStatusErrored) Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew) ERROR: retrieval failed: Retrieve: Retrieval Error: error generated by data transfer: deal data transfer failed: incomplete response

lotus client retrieve --miner f01278 bafykbzaceaj7ube4k2vhniqgdb6vq7ggcoebf4c6punse7ike2almme6esdxi mysql-2018-11-01.tar.gz.partcy

Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew) Recv: 0 B, Paid 0 FIL, ClientEventDealProposed (DealStatusWaitForAcceptance) Recv: 0 B, Paid 0 FIL, ClientEventUnsealPaymentRequested (DealStatusAccepted) Recv: 0 B, Paid 0 FIL, ClientEventPaymentChannelCreateInitiated (DealStatusPaymentChannelCreating) Recv: 0 B, Paid 0 FIL, ClientEventPaymentChannelReady (DealStatusPaymentChannelAllocatingLane) Recv: 0 B, Paid 0 FIL, ClientEventLaneAllocated (DealStatusOngoing) Recv: 0 B, Paid 0 FIL, ClientEventPaymentRequested (DealStatusFundsNeeded) Recv: 0 B, Paid 0 FIL, ClientEventSendFunds (DealStatusSendFunds) Recv: 0 B, Paid 0.000000000000000002 FIL, ClientEventPaymentSent (DealStatusOngoing) Recv: 0 B, Paid 0.000000000000000002 FIL, ClientEventDataTransferError (DealStatusErrored) Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew) ERROR: retrieval failed: Retrieve: Retrieval Error: error generated by data transfer: deal data transfer failed: incomplete response

Example 4 - miner is not accepting online retrieval deals lotus client retrieve --miner f047419 baga6ea4seaqgsbrsupr6az5zf2yjqai5t4xtgn7lf2zd62wnxxq2mgyhlvbxyda publicdomainmovies.tar.05

Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew) Recv: 0 B, Paid 0 FIL, ClientEventDealProposed (DealStatusWaitForAcceptance) Recv: 0 B, Paid 0 FIL, ClientEventDealRejected (DealStatusRetryLegacy) Recv: 0 B, Paid 0 FIL, ClientEventDealProposed (DealStatusWaitForAcceptanceLegacy) Recv: 0 B, Paid 0 FIL, ClientEventDealRejected (DealStatusRejected) Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew) ERROR: retrieval failed: Retrieve: Retrieval Proposal Rejected: deal rejected: miner is not accepting online retrieval deals

Example 5 - normal shutdown of state machine This provider is able to serve my retrieval for a 8GB file, but not 32GB ones.

lotus client retrieve --miner f022352 baga6ea4seaqoyyc52ajdq2v7723d3uixeem5ulwvbtv7h45776vrrg53nnf2gjq trusted_setup_phase2.tar.47

Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew) Recv: 0 B, Paid 0 FIL, ClientEventDealProposed (DealStatusWaitForAcceptance) Recv: 0 B, Paid 0 FIL, ClientEventDealAccepted (DealStatusAccepted) Recv: 0 B, Paid 0 FIL, ClientEventPaymentChannelAddingFunds (DealStatusPaymentChannelAllocatingLane) Recv: 0 B, Paid 0 FIL, ClientEventLaneAllocated (DealStatusOngoing) Recv: 0 B, Paid 0 FIL, ClientEventProviderCancelled (DealStatusCancelling) Recv: 0 B, Paid 0 FIL, ClientEventDataTransferError (DealStatusErrored) Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew) ERROR: retrieval failed: Retrieve: Retrieval Error: error generated by data transfer: unable to send cancel to channel FSM: normal shutdown of state machine

lotus version 1.4.1+git.d6c06881e

Importing chain ends with chain validation failed

Running lotus daemon --import-chain lotus daemon --import-chain minimal_finality_stateroots_336340_2020-12-19_17-00-00.car

v1.4.0 ends with:

 15.54 GiB / 15.58 GiB [======================================================================================================================================]  99.74% 115.68 MiB/s2020-12-21T07:06:20.325Z        INFO    badgerbs        [email protected]/levels.go:962      LOG Compact 1->2, del 3 tables, add 3 tables, took 658.043122ms

2020-12-21T07:06:20.325Z        INFO    badgerbs        [email protected]/levels.go:1010     [Compactor: 0] Compaction for level: 1 DONE
2020-12-21T07:06:20.325Z        INFO    badgerbs        [email protected]/levels.go:1000     [Compactor: 0] Running compaction: {level:1 score:1.5613754019141197 dropPrefixes:[]} for level: 1

 15.58 GiB / 15.58 GiB [================================================================================================================================] 100.00% 115.67 MiB/s 2m17s
2020-12-21T07:06:20.609Z        INFO    chainstore      store/store.go:527      clearing block validation cache...
2020-12-21T07:06:20.609Z        INFO    chainstore      store/store.go:566      0 block validation entries cleared.
2020-12-21T07:06:22.436Z        INFO    badgerbs        [email protected]/levels.go:962      LOG Compact 1->2, del 8 tables, add 8 tables, took 2.110439345s

2020-12-21T07:06:22.436Z        INFO    badgerbs        [email protected]/levels.go:1010     [Compactor: 0] Compaction for level: 1 DONE
2020-12-21T07:06:22.436Z        INFO    badgerbs        [email protected]/levels.go:1000     [Compactor: 0] Running compaction: {level:1 score:1.301148410886526 dropPrefixes:[]} for level: 1

2020-12-21T07:06:23.041Z        INFO    badgerbs        [email protected]/levels.go:962      LOG Compact 1->2, del 2 tables, add 2 tables, took 604.994671ms

2020-12-21T07:06:23.041Z        INFO    badgerbs        [email protected]/levels.go:1010     [Compactor: 0] Compaction for level: 1 DONE
2020-12-21T07:06:23.041Z        INFO    badgerbs        [email protected]/levels.go:1000     [Compactor: 0] Running compaction: {level:1 score:1.0409185104072094 dropPrefixes:[]} for level: 1

2020-12-21T07:06:24.137Z        INFO    badgerbs        [email protected]/levels.go:962      LOG Compact 1->2, del 4 tables, add 4 tables, took 1.09613443s

2020-12-21T07:06:24.137Z        INFO    badgerbs        [email protected]/levels.go:1010     [Compactor: 0] Compaction for level: 1 DONE
2020-12-21T07:06:42.565Z        WARN    chainstore      store/store.go:508      no heaviest tipset found, using [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2]
2020-12-21T07:06:42.565Z        INFO    chainstore      store/store.go:513      New heaviest tipset! [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2] (height=0)
2020-12-21T07:06:42.566Z        INFO    main    lotus/daemon.go:470     validating imported chain...
2020-12-21T07:07:04.375Z        INFO    statemgr        stmgr/stmgr.go:878      computing state (height: 0, ts=[bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2])
2020-12-21T07:07:04.375Z        INFO    statemgr        stmgr/stmgr.go:878      computing state (height: 1, ts=[bafy2bzacechdx6xd62lcyy7rnyc4uxcxhuwqslcxfvj77fxlwafij3nhzchpy])
2020-12-21T07:07:04.375Z        WARN    chainstore      store/store.go:485      reorgWorker quit
2020-12-21T07:07:04.421Z        INFO    badgerbs        [email protected]/db.go:1030 Storing value log head: {Fid:21 Len:33 Offset:513553269}

2020-12-21T07:07:04.574Z        INFO    badgerbs        [email protected]/levels.go:1000     [Compactor: 173] Running compaction: {level:0 score:1.73 dropPrefixes:[]} for level: 0

2020-12-21T07:07:05.444Z        INFO    badgerbs        [email protected]/levels.go:962      LOG Compact 0->1, del 4 tables, add 4 tables, took 870.449062ms

2020-12-21T07:07:05.445Z        INFO    badgerbs        [email protected]/levels.go:1010     [Compactor: 173] Compaction for level: 0 DONE
2020-12-21T07:07:05.445Z        INFO    badgerbs        [email protected]/db.go:553  Force compaction on level 0 done
ERROR: chain validation failed: getting block messages for tipset: failed to get messages for block: failed to load msgmeta (bafy2bzacecmwp4imjqhdg2zvc7j2s4xxahnn5jnudtrt335re24i4zim7ccfi): blockstore: block not found

[Thread] Documentation Requests

If you have a request for documentation on lotus, lotus-miner, or any related ecosystem tooling, please leave a comment here with what you want documented.

An example of a good request would be:

I would like documentation on how to operate multiple lotus-workers on different machines on my local network, with different machines for different jobs

or

I would like some documentation on how to examine exactly why my message failed on chain.

Please read through existing requests, and thumbs up any requests you also want (think of it like voting) instead of posting a duplicate (though if you want to add something to an existing request, please link to it in your comment, and let us know what you want to add)

Available sectors get upgraded to UpdateReplica sectors with no deals, when deals are ready to be added to sectors

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
[X] This is not a new feature request. If it is, please file a feature request instead.
[X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[ ] I did not make any code changes to lotus.

Lotus component

[ ] lotus daemon - chain sync
[ ] lotus miner - mining and block production
[X] lotus miner/worker - sealing
[ ] lotus miner - proving(WindowPoSt)
[ ] lotus miner/market - storage deal
[ ] lotus miner/market - retrieval deal
[ ] lotus miner/market - data transfer
[ ] lotus client
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[ ] Other

Lotus Version

lotus v1.15.2

Describe the Bug

Our sofiaminer has been configured with:

  MaxWaitDealsSectors = 2
  MaxSealingSectors = 2
  MaxSealingSectorsForDeals = 2

Today we noticed that 4 deals (each ~14GiB) were blocked in Adding to sector state, and sealing pipeline was blocked with 2 sectors:

957   UpdateReplica    YES      YES     3374050 (in 1 year 24 weeks)     CC
1007  UpdateReplica    YES      YES     3374170 (in 1 year 24 weeks)     CC

Those two sectors got created from Available sectors and their expiration epochs are after the 4 deals end epochs (i.e. the deals fit in these 2 sectors).

It is not clear why the sectors got into UpdateReplica states without the deals inside them.

After aborting the upgrade on these 2 sectors, 2 new sectors (from Available state) got upgraded, and successfully added the 4 deals to them and sealed properly into UpdateActivating state.

Logging Information

Broken UpdateReplica sectors with no deals:

2022-05-19T06:06:30.646+0300    INFO    sectors storage-sealing/input.go:278    Adding piece for deal 6328251 (publish msg: bafy2bzaceaajmdwdoik4u3gjmtfqhdoqpo57e5el6ac7akt4bupxclskcxkzs)
2022-05-19T06:06:30.650+0300    ERROR   sectors storage-sealing/input.go:514    we are trying to create a new sector with open sectors map[]
2022-05-19T06:06:30.651+0300    INFO    sectors storage-sealing/input.go:674    new deal sector decision        {"sealing": 0, "maxSeal": 2, "maxUpgrade": 2, "preferNew": false, "canCreate": true, "canUpgrade": true, "shouldUpgrade": true}
2022-05-19T06:06:30.665+0300    INFO    sectors storage-sealing/input.go:278    Adding piece for deal 6328250 (publish msg: bafy2bzaceaajmdwdoik4u3gjmtfqhdoqpo57e5el6ac7akt4bupxclskcxkzs)
2022-05-19T06:06:31.185+0300    INFO    sectors storage-sealing/input.go:620    Upgrading sector        {"number": "1007", "type": "deal", "proofType": 8, "expiration": "3374170", "pledge": "0.149331217034391605 FIL"}
2022-05-19T06:06:31.185+0300    ERROR   sectors storage-sealing/input.go:514    we are trying to create a new sector with open sectors map[]
2022-05-19T06:06:31.191+0300    INFO    sectors storage-sealing/input.go:144    starting to seal deal sector    {"sector": "1007", "trigger": "wait-timeout"}
2022-05-19T06:06:31.197+0300    INFO    sectors storage-sealing/states_sealing.go:48    performing filling up rest of the sector...     {"sector": "1007"}
2022-05-19T06:06:31.198+0300    WARN    sectors storage-sealing/states_sealing.go:72    Creating 1 filler pieces for sector 1007
2022-05-19T06:06:31.198+0300    INFO    sectors storage-sealing/states_sealing.go:88    Pledge {{127896 1007} 8}, contains []
2022-05-19T06:06:31.198+0300    DEBUG   advmgr  sector-storage/sched.go:361     SCHED 1 queued; 4 open windows
2022-05-19T06:06:31.200+0300    DEBUG   advmgr  sector-storage/sched.go:457     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]}]
2022-05-19T06:06:31.200+0300    DEBUG   advmgr  sector-storage/sched.go:458     SCHED Acceptable win: [[0 1]]
2022-05-19T06:06:31.200+0300    DEBUG   advmgr  sector-storage/sched.go:480     SCHED try assign sqi:0 sector 1007 to window 0 (awi:0)
2022-05-19T06:06:31.200+0300    DEBUG   advmgr  sector-storage/sched.go:480     SCHED try assign sqi:0 sector 1007 to window 1 (awi:1)
2022-05-19T06:06:31.200+0300    DEBUG   advmgr  sector-storage/sched.go:521     SCHED ASSIGNED  {"sqi": 0, "sector": "1007", "task": "seal/v0/addpiece", "window": 0, "worker": "80b730ae-31c9-4830-b176-b828a0df9dbe", "utilization": 0.2852374219200975}
2022-05-19T06:06:31.200+0300    DEBUG   advmgr  sector-storage/sched_worker.go:369      assign worker sector 1007
2022-05-19T06:06:31.200+0300    DEBUG   advmgr  sector-storage/sched.go:361     SCHED 0 queued; 4 open windows
2022-05-19T06:06:31.201+0300    DEBUG   advmgr  sector-storage/sched_worker.go:274      task done       {"workerid": "80b730ae-31c9-4830-b176-b828a0df9dbe"}
2022-05-19T06:06:31.201+0300    DEBUG   advmgr  sector-storage/sched.go:361     SCHED 0 queued; 4 open windows
2022-05-19T06:07:34.664+0300    ERROR   sectors storage-sealing/fsm.go:34       unhandled sector error (1007): checkPieces sanity check error:
    github.com/filecoin-project/lotus/extern/storage-sealing.handleErrors
        /root/lotus/extern/storage-sealing/states_replica_update.go:295
  - sector 1007 must have deals, but does not

2022-05-19T08:28:59.970+0300    INFO    sectors storage-sealing/input.go:278    Adding piece for deal 6333497 (publish msg: bafy2bzaceawz6tlxavlwdjiynfhsp2lrdnfeo66phthy42jaorhc5tbctalog)
2022-05-19T08:28:59.972+0300    ERROR   sectors storage-sealing/input.go:514    we are trying to create a new sector with open sectors map[]
2022-05-19T08:28:59.973+0300    INFO    sectors storage-sealing/input.go:674    new deal sector decision        {"sealing": 1, "maxSeal": 2, "maxUpgrade": 2, "preferNew": false, "canCreate": true, "canUpgrade": true, "shouldUpgrade": true}
2022-05-19T08:28:59.995+0300    INFO    sectors storage-sealing/input.go:278    Adding piece for deal 6333498 (publish msg: bafy2bzaceawz6tlxavlwdjiynfhsp2lrdnfeo66phthy42jaorhc5tbctalog)
2022-05-19T08:29:00.567+0300    INFO    sectors storage-sealing/input.go:620    Upgrading sector        {"number": "957", "type": "deal", "proofType": 8, "expiration": "3374050", "pledge": "0.14650541070944655 FIL"}
2022-05-19T08:29:00.568+0300    ERROR   sectors storage-sealing/input.go:514    we are trying to create a new sector with open sectors map[]
2022-05-19T08:29:00.573+0300    INFO    sectors storage-sealing/input.go:144    starting to seal deal sector    {"sector": "957", "trigger": "wait-timeout"}
2022-05-19T08:29:00.579+0300    INFO    sectors storage-sealing/states_sealing.go:48    performing filling up rest of the sector...     {"sector": "957"}
2022-05-19T08:29:00.579+0300    WARN    sectors storage-sealing/states_sealing.go:72    Creating 1 filler pieces for sector 957
2022-05-19T08:29:00.579+0300    INFO    sectors storage-sealing/states_sealing.go:88    Pledge {{127896 957} 8}, contains []
2022-05-19T08:29:00.579+0300    DEBUG   advmgr  sector-storage/sched.go:361     SCHED 1 queued; 4 open windows
2022-05-19T08:29:00.582+0300    DEBUG   advmgr  sector-storage/sched.go:457     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]}]
2022-05-19T08:29:00.582+0300    DEBUG   advmgr  sector-storage/sched.go:458     SCHED Acceptable win: [[0 3]]
2022-05-19T08:29:00.582+0300    DEBUG   advmgr  sector-storage/sched.go:480     SCHED try assign sqi:0 sector 957 to window 0 (awi:0)
2022-05-19T08:29:00.582+0300    DEBUG   advmgr  sector-storage/sched.go:480     SCHED try assign sqi:0 sector 957 to window 3 (awi:1)
2022-05-19T08:29:00.582+0300    DEBUG   advmgr  sector-storage/sched.go:521     SCHED ASSIGNED  {"sqi": 0, "sector": "957", "task": "seal/v0/addpiece", "window": 0, "worker": "80b730ae-31c9-4830-b176-b828a0df9dbe", "utilization": 0.2852374219200975}
2022-05-19T08:29:00.582+0300    DEBUG   advmgr  sector-storage/sched_worker.go:369      assign worker sector 957
2022-05-19T08:29:00.582+0300    DEBUG   advmgr  sector-storage/sched.go:361     SCHED 0 queued; 4 open windows
2022-05-19T08:29:00.582+0300    DEBUG   advmgr  sector-storage/sched_worker.go:274      task done       {"workerid": "80b730ae-31c9-4830-b176-b828a0df9dbe"}
2022-05-19T08:29:00.582+0300    DEBUG   advmgr  sector-storage/sched.go:361     SCHED 0 queued; 4 open windows
2022-05-19T08:29:06.011+0300    INFO    miner   miner/miner.go:479      completed mineOne       {"tookMilliseconds": 7, "forRound": 1821059, "baseEpoch": 1821058, "baseDeltaSeconds": 6, "nullRounds": 0, "lateStart": false, "beaconEpoch": 1916903, "lookbackEpochs": 900, "networkPowerAtLookback": "19312307760272900096", "minerPowerAtLookback": "63971695689728", "isEligible": true, "isWinner": false, "error": null}
2022-05-19T08:30:04.770+0300    ERROR   sectors storage-sealing/fsm.go:34       unhandled sector error (957): checkPieces sanity check error:
    github.com/filecoin-project/lotus/extern/storage-sealing.handleErrors
        /root/lotus/extern/storage-sealing/states_replica_update.go:295
  - sector 957 must have deals, but does not

I trigger abort upgrade on the 2 sectors from above, and the deals got added to the new sectors (around 13:00 timestamp):

2022-05-19T12:57:00.015+0300    INFO    sectors storage-sealing/input.go:726    aborting upgrade of sector      {"sector": "957", "trigger": "user"}
2022-05-19T12:57:00.015+0300    WARN    sectors storage-sealing/fsm.go:763      sector 957 got error event sealing.SectorAbortUpgrade: triggered by user
2022-05-19T12:57:00.021+0300    ERROR   sectors storage-sealing/input.go:514    we are trying to create a new sector with open sectors map[]
2022-05-19T12:57:00.022+0300    INFO    sectors storage-sealing/input.go:674    new deal sector decision        {"sealing": 1, "maxSeal": 2, "maxUpgrade": 2, "preferNew": false, "canCreate": true, "canUpgrade": true, "shouldUpgrade": true}
2022-05-19T12:57:00.592+0300    INFO    sectors storage-sealing/input.go:620    Upgrading sector        {"number": "978", "type": "deal", "proofType": 8, "expiration": "3374050", "pledge": "0.146720918263824136 FIL"}
2022-05-19T12:57:00.599+0300    ERROR   sectors storage-sealing/input.go:514    we are trying to create a new sector with open sectors map[{127896 978}:0xc01d2c3520]
2022-05-19T12:57:00.600+0300    INFO    sectors storage-sealing/input.go:674    new deal sector decision        {"sealing": 2, "maxSeal": 2, "maxUpgrade": 2, "preferNew": false, "canCreate": false, "canUpgrade": false, "shouldUpgrade": false}
2022-05-19T12:57:00.600+0300    DEBUG   advmgr  sector-storage/sched.go:361     SCHED 1 queued; 4 open windows
2022-05-19T12:57:00.602+0300    DEBUG   advmgr  sector-storage/sched.go:457     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]}]
2022-05-19T12:57:00.602+0300    DEBUG   advmgr  sector-storage/sched.go:458     SCHED Acceptable win: [[2 3]]
2022-05-19T12:57:00.602+0300    DEBUG   advmgr  sector-storage/sched.go:480     SCHED try assign sqi:0 sector 978 to window 2 (awi:0)
2022-05-19T12:57:00.603+0300    DEBUG   advmgr  sector-storage/sched.go:480     SCHED try assign sqi:0 sector 978 to window 3 (awi:1)
2022-05-19T12:57:00.603+0300    DEBUG   advmgr  sector-storage/sched.go:521     SCHED ASSIGNED  {"sqi": 0, "sector": "978", "task": "seal/v0/addpiece", "window": 2, "worker": "80b730ae-31c9-4830-b176-b828a0df9dbe", "utilization": 0.2852374219200975}
2022-05-19T12:57:00.603+0300    DEBUG   advmgr  sector-storage/sched_worker.go:369      assign worker sector 978
2022-05-19T12:57:00.603+0300    DEBUG   advmgr  sector-storage/sched.go:361     SCHED 0 queued; 4 open windows
2022-05-19T12:57:00.603+0300    DEBUG   advmgr  sector-storage/sched_worker.go:274      task done       {"workerid": "80b730ae-31c9-4830-b176-b828a0df9dbe"}
2022-05-19T12:57:00.603+0300    DEBUG   advmgr  sector-storage/sched.go:361     SCHED 0 queued; 4 open windows
2022-05-19T12:57:02.688+0300    INFO    sectors storage-sealing/input.go:726    aborting upgrade of sector      {"sector": "1007", "trigger": "user"}
2022-05-19T12:57:02.688+0300    WARN    sectors storage-sealing/fsm.go:763      sector 1007 got error event sealing.SectorAbortUpgrade: triggered by user
2022-05-19T12:57:02.696+0300    ERROR   sectors storage-sealing/input.go:514    we are trying to create a new sector with open sectors map[{127896 978}:0xc01d2c3520]
2022-05-19T12:57:02.697+0300    INFO    sectors storage-sealing/input.go:674    new deal sector decision        {"sealing": 1, "maxSeal": 2, "maxUpgrade": 2, "preferNew": false, "canCreate": true, "canUpgrade": true, "shouldUpgrade": true}
2022-05-19T12:57:03.201+0300    INFO    sectors storage-sealing/input.go:620    Upgrading sector        {"number": "875", "type": "deal", "proofType": 8, "expiration": "3374050", "pledge": "0.146775113865666972 FIL"}

Logs from `lotus-worker`

2022-05-19T06:06:31.211+0300    DEBUG   advmgr  sector-storage/worker_local.go:158      acquired sector {{127896 1007} 8} (e:0; a:1): {{0 0} /root/.lotusworker/unsealed/s-t0127896-1007    }
2022-05-19T06:06:34.205+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -12288
2022-05-19T06:06:44.206+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -12288
2022-05-19T06:06:54.208+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -12288
2022-05-19T06:07:04.209+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -32768
2022-05-19T06:07:14.218+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -36864
2022-05-19T06:07:24.220+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -36864
2022-05-19T06:07:34.221+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -40960
2022-05-19T08:29:00.587+0300    DEBUG   advmgr  sector-storage/worker_local.go:158      acquired sector {{127896 957} 8} (e:0; a:1): {{0 0} /root/.lotusworker/unsealed/s-t0127896-957    }
2022-05-19T08:29:05.281+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -16384
2022-05-19T08:29:15.282+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -16384
2022-05-19T08:29:25.283+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -16384
2022-05-19T08:29:35.283+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -32768
2022-05-19T08:29:45.298+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -49152
2022-05-19T08:29:55.302+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -49152


2022-05-19T12:57:00.609+0300    DEBUG   advmgr  sector-storage/worker_local.go:158      acquired sector {{127896 978} 8} (e:0; a:1): {{0 0} /root/.lotusworker/unsealed/s-t0127896-978    }
2022-05-19T12:57:03.221+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -28672
2022-05-19T12:57:03.221+0300    DEBUG   advmgr  sector-storage/worker_local.go:158      acquired sector {{127896 875} 8} (e:0; a:1): {{0 0} /root/.lotusworker/unsealed/s-t0127896-875    }
2022-05-19T12:57:07.218+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -73728
2022-05-19T12:57:17.219+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -77824
2022-05-19T12:57:27.221+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -86016
2022-05-19T12:57:37.222+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -86016
2022-05-19T12:57:47.223+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -86016                                                                     2022-05-19T12:57:57.224+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -86016
2022-05-19T12:58:07.226+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -86016
2022-05-19T12:58:17.227+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -86016                                                                     2022-05-19T12:58:27.228+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -86016
2022-05-19T12:58:37.229+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -94208
2022-05-19T12:58:47.230+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -106496
2022-05-19T12:58:57.231+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -106496
2022-05-19T12:59:07.232+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -106496
2022-05-19T12:59:17.233+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -110592
2022-05-19T12:59:27.237+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -110592
2022-05-19T12:59:37.238+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -110592
2022-05-19T12:59:47.239+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -110592
2022-05-19T12:59:57.240+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -110592
2022-05-19T13:00:07.241+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -110592                                                                    2022-05-19T13:00:17.245+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -110592
2022-05-19T13:00:27.246+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=68719476736, reserved: -110592
2022-05-19T13:00:31.235+0300    DEBUG   advmgr  sector-storage/worker_local.go:158      acquired sector {{127896 978} 8} (e:1; a:0): {{0 0} /root/.lotusworker/unsealed/s-t0127896-978    }                     2022-05-19T13:00:37.247+0300    WARN    stores  stores/local.go:130     negative reserved storage: p.reserved=34359738368, reserved: -53248
2022-05-19T13:00:45.312+0300    DEBUG   advmgr  sector-storage/worker_local.go:158      acquired sector {{127896 875} 8} (e:1; a:0): {{0 0} /root/.lotusworker/unsealed/s-t0127896-875    }
2022-05-19T13:04:18.534+0300    INFO    stores  stores/http_handler.go:189      SERVE Alloc check /remote/unsealed/s-t0127896-978/8/allocated/0/17045651456
2022-05-19T13:04:18.535+0300    DEBUG   stores  stores/http_handler.go:283      returning ok: worker has unsealed file with unsealed piece, sector:{Miner:127896 Number:978}, offset:0, size:17045651456
2022-05-19T13:04:18.539+0300    INFO    stores  stores/http_handler.go:189      SERVE Alloc check /remote/unsealed/s-t0127896-978/8/allocated/0/17045651456
2022-05-19T13:04:18.540+0300    DEBUG   stores  stores/http_handler.go:283      returning ok: worker has unsealed file with unsealed piece, sector:{Miner:127896 Number:978}, offset:0, size:17045651456
2022-05-19T13:04:18.606+0300    DEBUG   stores  stores/http_handler.go:157      served sector file/dir, sectorID={Miner:127896 Number:978}, fileType=unsealed, path=/root/.lotusworker/unsealed/s-t0127896-978
2022-05-19T13:04:19.409+0300    INFO    stores  stores/http_handler.go:189      SERVE Alloc check /remote/unsealed/s-t0127896-978/8/allocated/17045651456/17045651456
2022-05-19T13:04:19.410+0300    DEBUG   stores  stores/http_handler.go:283      returning ok: worker has unsealed file with unsealed piece, sector:{Miner:127896 Number:978}, offset:17045651456,

Repo Steps

Ntwk/wallaby reset files
Related Issues

no issues to reference

Proposed Changes

new boostrap peers .pi file and new genesis car file

Additional Info

move presealed sectors, more geo distributed boostrappers

Checklist

@raulk @vyzo @Stebalien @jennijuju

Before you mark the PR ready for review, please make sure that:

[x] Commits have a clear commit message.

[ ] PR title is in the form of of <PR type>: <area>: <change being made>

example: fix: mempool: Introduce a cache for valid signatures

PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test

area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps

[ ] New features have usage guidelines and / or documentation updates in

[ ] Lotus Documentation

[ ] Discussion Tutorials

[ ] Tests exist for new functionality or change in behavior

[ ] CI is green
feat: blockstore: Envvar can adjust badger compaction worker poolsize
Related Issues

This is a proposal for closing #9972. Happy to add docs or tests as needed.

Proposed Changes

Include envvar LOTUS_CHAIN_BADGERSTORE_COMPACTIONWORKERNUM which accepts a non-negative int that configures the number of compaction workers for the badger blockstore

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

[x] Commits have a clear commit message.

[x] PR title is in the form of of <PR type>: <area>: <change being made>

example: fix: mempool: Introduce a cache for valid signatures

PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test

area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps

[x] New features have usage guidelines and / or documentation updates in

[x] Lotus Documentation (PR: https://github.com/filecoin-project/lotus-docs/pull/463) project/lotus/discussions/categories/tutorials)

[ ] Tests exist for new functionality or change in behavior

[x] CI is green
Allow number of BadgerDB Compaction workers to be adjusted via environment variable
Checklist

[X] This is not a new feature or an enhancement to the Filecoin protocol. If it is, please open an FIP issue.

[X] This is not a new feature request. If it is, please file a feature request instead.

[X] This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.

[X] I have a specific, actionable, and well motivated improvement to propose.

Lotus component

[X] lotus daemon - chain sync

[ ] lotus miner - mining and block production

[ ] lotus miner/worker - sealing

[ ] lotus miner - proving(WindowPoSt)

[ ] lotus miner/market - storage deal

[ ] lotus miner/market - retrieval deal

[ ] lotus miner/market - data transfer

[ ] lotus client

[ ] lotus JSON-RPC API

[ ] lotus message management (mpool)

[ ] Other

Improvement Suggestion

Sentinel has a use case for Lily where it can import a snapshot for read-only indexing purposes. This results in a badger blockstore does not intend to be updated further. When importing such a snapshot, a long compaction process occurs on the blockstore which blocks the quick availability of the repo for indexing. As an operator, I would like to disable this compaction in certain (all of?) Sentinel's use cases to decrease the wait until the node can become ready.

To this end, I would like to use some interface to manipulate the badger configuration for the number of compactors used during runtime.

Open Questions: Is this a bad idea? Why? Are there other details I'm overlooking?
Eth JSON-RPC: use ToFilecoinAddress() to get f4 addr and validate v sig
Related Issues

https://github.com/filecoin-project/ref-fvm/issues/1030 https://github.com/filecoin-project/ref-fvm/issues/1335

Proposed Changes

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

[ ] Commits have a clear commit message.

[ ] PR title is in the form of of <PR type>: <area>: <change being made>

example: fix: mempool: Introduce a cache for valid signatures

PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test

area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps

[ ] New features have usage guidelines and / or documentation updates in

[ ] Lotus Documentation

[ ] Discussion Tutorials

[ ] Tests exist for new functionality or change in behavior

[ ] CI is green
fix: cli: remove requirements in helptext
Proposed Changes

The current lotus-worker run help text outputs the hw-requirements for some of the sealing-tasks.

lotus-worker run -h NAME: lotus-worker run - Start lotus worker USAGE: lotus-worker run [command options] [arguments...] OPTIONS: --precommit1 enable precommit1 (32G sectors: 1 core, 128GiB Memory) (default: true) [$LOTUS_WORKER_PRECOMMIT1]

A lot of the requirements in this text is wrong, as for example the PC1 requires 64GiB of memory, and might also use up to a single core complex (1 main core, 3 producer threads) if the FIL_PROOFS_USE_MULTICORE_SDR=1 enviroment variable is exported. I propose that we remove the the sealing hw-requirements from this helptext as there are a lot of nuances that is hard to communicate in the space available in the cli, and rather communicate them in the documentation https://lotus.filecoin.io/storage-providers/get-started/hardware-requirements/ - which has now been updated (only missing PRU/PRU2, but will update those hw-requirements on Monday):

Checklist

Before you mark the PR ready for review, please make sure that:

[x] Commits have a clear commit message.

[x] PR title is in the form of of <PR type>: <area>: <change being made>

example: fix: mempool: Introduce a cache for valid signatures

PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test

area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps

[ ] New features have usage guidelines and / or documentation updates in

[ ] Lotus Documentation

[ ] Discussion Tutorials

[ ] Tests exist for new functionality or change in behavior

[ ] CI is green

Implementation of the Filecoin protocol, written in Go

Project Lotus - 莲

Building & Documentation

Reporting a Vulnerability

Related packages

Contribute

License

Owner

Filecoin

Comments

Arm64 Raspberry Pi Build Failure

ChainGetTipSetByHeight method synchronization lacks message

updating to new datastore/blockstore code with contexts

Status

Mined blocks became orphan blocks

1.18.0, the miner process crashes during execution of the AddPiece phase

Checklist

Lotus component

Lotus Version

Describe the Bug

Logging Information

Repo Steps

GPU stuck at P2

Lotus-bench results thread (v20 params)

Cannot perform any retrieval for 32GB files

Importing chain ends with chain validation failed

[Thread] Documentation Requests

Available sectors get upgraded to UpdateReplica sectors with no deals, when deals are ready to be added to sectors

Checklist

Lotus component

Lotus Version

Describe the Bug

Logging Information

Logs from lotus-worker

Repo Steps

Ntwk/wallaby reset files

Related Issues

Proposed Changes

Additional Info

Checklist

feat: blockstore: Envvar can adjust badger compaction worker poolsize

Related Issues

Proposed Changes

Additional Info

Checklist

Allow number of BadgerDB Compaction workers to be adjusted via environment variable

Checklist

Lotus component

Improvement Suggestion

Eth JSON-RPC: use ToFilecoinAddress() to get f4 addr and validate v sig

Related Issues

Proposed Changes

Additional Info

Checklist

fix: cli: remove requirements in helptext

Proposed Changes

Checklist

Related tags

A Go client and CLI for Filecoin Storage Auctions.

Filecoin sector recover

Yet another filecoin secondary retrieval client

Go language implementation of a blockchain based on the BDLS BFT protocol. The implementation was adapted from Ethereum and Sperax implementation

Eunomia is a distributed application framework that support Gossip protocol, QuorumNWR algorithm, PBFT algorithm, PoW algorithm, and ZAB protocol and so on.

Interblockchain communication protocol (IBC) implementation in Golang.

Go Implementation of the Spacemesh protocol full node. 💾⏰💪

Official Golang implementation of the Ethereum protocol

Security research and open source implementation of the Apple 'Wireless Accessory Configuration' (WAC) protocol

Official Go implementation of the Ethereum protocol

RepoETH - Official Golang implementation of the Ethereum protocol

Go-ethereum - Official Golang implementation of the Ethereum protocol

Dxc - Go implementation of DxChain3.0 protocol

Official Golang implementation of the Ethereum protocol

Koisan-chain - Official Golang implementation of the Koisan protocol

Ethereum go-ethereum - Official Golang implementation of the Ethereum protocol

Terra client in golang with multiple protocol implementation (anchor, astroport, prism, ...)

This is a close to decentralized RSS3 Network implementation of RSS3 protocol v0.4.0 with full indexing function in Go

Logs from `lotus-worker`