WindowPost failed after lotus-miner init restore #9724

MMNNMMNM · 2022-11-18T11:19:07Z

MMNNMMNM
Nov 18, 2022

Checklist

This is not a security-related bug/issue. If it is, please follow please follow the security policy.
This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
This is not a new feature request. If it is, please file a feature request instead.
This is not an enhancement request. If it is, please file a improvement suggestion instead.
I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
I did not make any code changes to lotus.

Lotus component

Lotus Version

Lotus
	Daemon:  1.17.2+mainnet+git.fb0fb7144.dirty+api1.5.0
	Local: lotus version 1.17.2+mainnet+git.fb0fb7144.dirty

	Lotus-miner
	Daemon:  1.17.2+mainnet+git.fb0fb7144.dirty+api1.5.0
	Local: lotus-miner version 1.17.2+mainnet+git.fb0fb7144.dirty

Describe the Bug

lotus-miner info

Enabled subsystems (from miner API): [Mining Sealing SectorStorage Markets]
Enabled subsystems (from markets API): [Mining Sealing SectorStorage Markets]
Chain: [sync ok] [basefee 312.905 pFIL]
Miner: f01393827 (32 GiB sectors)
Power: 0  / 18.8 Ei (0.0000%)
        Raw: 0 B / 15.82 EiB (0.0000%)
        Committed: 1001 TiB
        Proving: 1001 TiB (1001 TiB Faulty, 100.00%)
Below minimum power threshold, no blocks will be won
Miner Balance:    5342.417 FIL
      PreCommit:  0
      Pledge:     4670.152 FIL
      Vesting:    511.699 FIL
      Available:  160.567 FIL
Market Balance:   773.302 μFIL
       Locked:    361.062 nFIL
       Available: 772.941 μFIL
Worker Balance:   999.272 mFIL
Total Spendable:  161.567 FIL

[detached from 1529.miner]

root@main_d:~# lotus-miner info

Enabled subsystems (from miner API): [Mining Sealing SectorStorage Markets]
Enabled subsystems (from markets API): [Mining Sealing SectorStorage Markets]
Chain: [sync ok] [basefee 282.71 pFIL]
Miner: f01393827 (32 GiB sectors)
Power: 0  / 18.8 Ei (0.0000%)
        Raw: 0 B / 15.82 EiB (0.0000%)
        Committed: 1001 TiB
        Proving: 1001 TiB (1001 TiB Faulty, 100.00%)
Below minimum power threshold, no blocks will be won
Miner Balance:    5342.417 FIL
      PreCommit:  0
      Pledge:     4670.152 FIL
      Vesting:    511.699 FIL
      Available:  160.567 FIL
Market Balance:   773.302 μFIL
       Locked:    361.062 nFIL
       Available: 772.941 μFIL
Worker Balance:   999.272 mFIL
Total Spendable:  161.567 FIL

Sectors:
        Total: 0

Workers: Seal(1) WdPoSt(0) WinPoSt(0)

Storage Deals: 0, 0 B

Retrieval Deals (complete): 0, 0 B

lotus-miner proving info

Miner: f01393827
Current Epoch:           2348785
Proving Period Boundary: 290
Proving Period Start:    2347490 (10 hours 47 minutes ago)
Next Period Start:       2350370 (in 13 hours 12 minutes)

Faults:      32040 (100.00%)
Recovering:  0
Deadline Index:       21
Deadline Sectors:     0
Deadline Open:        2348750 (17 minutes 30 seconds ago)
Deadline Close:       2348810 (in 12 minutes 30 seconds)
Deadline Challenge:   2348730 (27 minutes 30 seconds ago)
Deadline FaultCutoff: 2348680 (52 minutes 30 seconds ago)

lotus-miner proving deadlines

Miner: f01393827
deadline  partitions  sectors (faults)  proven partitions
0         1           2349 (2298)       0
1         1           2349 (2346)       0
2         1           2349 (2348)       0
3         1           2349 (2349)       0
4         1           2349 (2349)       0
5         1           2349 (2349)       0
6         1           2349 (2349)       0
7         1           2349 (2346)       0
8         1           2349 (2349)       0
9         1           2349 (2349)       0
10        1           2349 (2349)       0
11        1           2349 (2349)       0
12        1           2349 (2341)       0
13        1           1938 (1491)       0
14        1           83 (57)           0
15        1           39 (21)           0
16        0           0 (0)             0
17        0           0 (0)             0
18        0           0 (0)             0
19        0           0 (0)             0
20        0           0 (0)             0
21        0           0 (0)             0  (current)

First Sector Fault happened
Timestamp in log - 2022-11-13T09:15:29.999+0900
1. I try to run lotus-miner re-sync chain data
  Timestamp in log - 2022-11-13T12:14:54.639+0900 ~ 2022-11-14T14:16:43.117+0900
  
  At this time, lotus-miner run failed with "keystore not found"
3)lotus-miner init restore with backup file
Timestamp in log - 2022-11-14T15:47:20.031+0900
1. Update config.toml and storage.json with original files.
After restore backup file, we can check sectors list / status / find.
But window post was failed.
- 2022-11-15T09:15:34.947+0900 WARN advmgr sector-storage/faults.go:122 CheckProvable Sector FAULT: generating vanilla proof
I checked GPU is running when windowPost simulation. (lotus-miner proving compute)
Also I can see lotus-miner read local sectors when I restart lotus-miner.

Logging Information

2022-11-14T15:57:57.704+0900	WARN	miner	miner/miner.go:477	completed mineOne	{"tookMilliseconds": 4, "forRound": 2336756, "baseEpoch": 2336755, "baseDeltaSeconds": 27, "nullRounds": 0, "lateStart": true, "beaconEpoch": 0, "lookbackEpochs": 900, "networkPowerAtLookback": "21596441474685894656", "minerPowerAtLookback": "0", "isEligible": false, "isWinner": false, "error": null}
	 2022-11-14T15:57:57.705+0900	WARN	storageminer	storage/wdpost_run.go:237	Checked sectors	{"checked": 0, "good": 0}
	 2022-11-14T15:57:57.741+0900	INFO	net/identify	identify/id.go:364	failed negotiate identify protocol with peer	{"peer": "12D3KooWGLzAhMKGhohnG9j4hLPq7wdGbhXMvQKUpaHSd222PVvD", "error": "stream reset"}
	 2022-11-14T15:57:57.742+0900	INFO	net/identify	identify/id.go:364	failed negotiate identify protocol with peer	{"peer": "12D3KooWNrucYhEyoZQYSFByaU9FfuWtzJM5ozSJJ44VLBFCVaZU", "error": "stream reset"}
	 2022-11-14T15:57:57.749+0900	INFO	miner	miner/warmup.go:48	skipping winning PoSt warmup, no sectors
	 2022-11-14T15:57:57.998+0900	WARN	advmgr	sector-storage/faults.go:122	CheckProvable Sector FAULT: generating vanilla proof	{"sector": {"ID":{"Miner":1393827,"Number":30665},"ProofType":8}, "err": "generate_single_vanilla_proof: merkle_tree failed: SectorId(30665)\n\nCaused by:\n    Inconsistent tree data\n\nStack backtrace:\n   0: anyhow::error::<impl anyhow::Error>::msg\n   1: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::from_data_store\n   2: storage_proofs_core::merkle::builders::create_tree\n   3: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::merkle_tree\n   4: filecoin_proofs_api::post::generate_single_vanilla_proof_inner\n   5: filecoin_proofs_api::post::generate_single_vanilla_proof\n   6: std::panicking::try\n   7: filcrypto::util::types::catch_panic_response\n   8: generate_single_vanilla_proof\n   9: _cgo_2731a736ee7f_Cfunc_generate_single_vanilla_proof\n             at /tmp/go-build/cgo-gcc-prolog:347:11\n  10: runtime.asmcgocall\n             at /usr/local/go/src/runtime/asm_amd64.s:765"}
	 2022-11-14T15:57:57.999+0900	WARN	advmgr	sector-storage/faults.go:122	CheckProvable Sector FAULT: generating vanilla proof	{"sector": {"ID":{"Miner":1393827,"Number":32291},"ProofType":8}, "err": "generate_single_vanilla_proof: merkle_tree failed: SectorId(32291)\n\nCaused by:\n    Inconsistent tree data\n\nStack backtrace:\n   0: anyhow::error::<impl anyhow::Error>::msg\n

Repo Steps

Run '...'
Do '...'
See error '...'
...

Reiers · 2022-11-18T13:16:48Z

Reiers
Nov 18, 2022

Hi @MMNNMMNM

Thanks for the report. We need more information!

How old was the backup?
Does every deadline have good sectors?

lotus-miner proving check --slow --only-bad 0

It also looks like you are running Lotus with custom code or have made adjustment hence the .dirty
What changes has been made? If it's mistake, please upgrade Lotus to latest stable release v1.18.0.

1. cd lotus/
2. git pull
3. git checkout v1.18.0
4. git submodule update
5. make clean
6. RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA=1 make all
7. sudo make install

* remove FFI_USE_CUDA=1 if you are not using it.

Thank you !

1 reply

lbj2004032 Mar 20, 2023

I tried 1.20.4 on one of the nodes and was able to start successfully. I'm not sure if this version is fixing this issue

MMNNMMNM · 2022-11-19T06:38:53Z

MMNNMMNM
Nov 19, 2022
Author

Hi thanks for your answer!

How old was the backup?
- Sep-27th-2022
Does every deadline have good sectors?
- Deadline 0 has a total of 2298 sectors.
  If I run lotus-miner proving check --slow --only-bad 0, all bad sectors come out.
  I will show the log of one sector below.
  (The following is the log based on lotus v1.18.0 version.)

deadline  partition  sector  status
0         0          694     bad (generating vanilla proof: generate_single_vanilla_proof: vanilla_proof failed: SectorId(694)

Caused by:
    Generated vanilla proof for sector SectorId(694) is invalid

Stack backtrace:
   0: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
   1: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
   2: rayon::iter::plumbing::Producer::fold_with
   3: rayon::iter::plumbing::bridge_producer_consumer::helper
   4: std::panicking::try
   5: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
   6: rayon_core::registry::WorkerThread::wait_until_cold
   7: rayon_core::registry::in_worker
   8: rayon::iter::plumbing::bridge_producer_consumer::helper
   9: std::panicking::try
  10: rayon_core::registry::in_worker
  11: rayon::iter::plumbing::bridge_producer_consumer::helper
  12: std::panicking::try
  13: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  14: rayon_core::registry::WorkerThread::wait_until_cold
  15: rayon_core::registry::in_worker
  16: rayon::iter::plumbing::bridge_producer_consumer::helper
  17: std::panicking::try
  18: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  19: rayon_core::registry::WorkerThread::wait_until_cold
  20: rayon_core::registry::in_worker
  21: rayon::iter::plumbing::bridge_producer_consumer::helper
  22: rayon_core::job::StackJob<L,F,R>::run_inline
  23: rayon_core::registry::in_worker
  24: rayon::iter::plumbing::bridge_producer_consumer::helper
  25: rayon_core::job::StackJob<L,F,R>::run_inline
  26: rayon_core::registry::in_worker
  27: rayon::iter::plumbing::bridge_producer_consumer::helper
  28: std::panicking::try
  29: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  30: rayon_core::registry::WorkerThread::wait_until_cold
  31: rayon_core::registry::in_worker
  32: rayon::iter::plumbing::bridge_producer_consumer::helper
  33: std::panicking::try
  34: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  35: rayon_core::registry::WorkerThread::wait_until_cold
  36: rayon_core::registry::in_worker
  37: rayon::iter::plumbing::bridge_producer_consumer::helper
  38: <rayon::iter::plumbing::bridge::Callback<C> as rayon::iter::plumbing::ProducerCallback<I>>::callback
  39: rayon::iter::collect::special_extend
  40: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend
  41: merkletree::store::Store::build
  42: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::build_partial_tree
  43: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
  44: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
  45: <storage_proofs_core::merkle::tree::MerkleTreeWrapper<H,S,U,V,W> as storage_proofs_core::merkle::tree::MerkleTreeTrait>::gen_cached_proof
  46: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
  47: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
  48: rayon::iter::plumbing::Producer::fold_with
  49: rayon::iter::plumbing::bridge_producer_consumer::helper
  50: std::panicking::try
  51: rayon_core::registry::in_worker
  52: rayon::iter::plumbing::bridge_producer_consumer::helper
  53: std::panicking::try
  54: rayon_core::registry::in_worker
  55: rayon::iter::plumbing::bridge_producer_consumer::helper
  56: std::panicking::try
  57: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  58: std::panicking::try
  59: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  60: rayon_core::registry::WorkerThread::wait_until_cold
  61: rayon_core::registry::in_worker
  62: rayon::iter::plumbing::bridge_producer_consumer::helper
  63: rayon_core::job::StackJob<L,F,R>::run_inline
  64: rayon_core::registry::in_worker
  65: rayon::iter::plumbing::bridge_producer_consumer::helper
  66: <rayon::iter::plumbing::bridge::Callback<C> as rayon::iter::plumbing::ProducerCallback<I>>::callback
  67: rayon::iter::collect::special_extend
  68: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend
  69: merkletree::store::Store::build
  70: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::build_partial_tree
  71: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
  72: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
  73: <storage_proofs_core::merkle::tree::MerkleTreeWrapper<H,S,U,V,W> as storage_proofs_core::merkle::tree::MerkleTreeTrait>::gen_cached_proof
  74: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
  75: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
  76: rayon::iter::plumbing::Producer::fold_with
  77: rayon::iter::plumbing::bridge_producer_consumer::helper
  78: std::panicking::try
  79: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  80: rayon_core::registry::WorkerThread::wait_until_cold
  81: rayon_core::registry::in_worker
  82: rayon::iter::plumbing::bridge_producer_consumer::helper
  83: std::panicking::try
  84: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  85: rayon_core::registry::WorkerThread::wait_until_cold
  86: rayon_core::registry::in_worker
  87: rayon::iter::plumbing::bridge_producer_consumer::helper
  88: <rayon::iter::plumbing::bridge::Callback<C> as rayon::iter::plumbing::ProducerCallback<I>>::callback
  89: rayon::iter::collect::special_extend
  90: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend
  91: merkletree::store::Store::build
  92: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::build_partial_tree
  93: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
  94: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
  95: <storage_proofs_core::merkle::tree::MerkleTreeWrapper<H,S,U,V,W> as storage_proofs_core::merkle::tree::MerkleTreeTrait>::gen_cached_proof
  96: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
  97: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
  98: rayon::iter::plumbing::Producer::fold_with
  99: rayon::iter::plumbing::bridge_producer_consumer::helper
 100: std::panicking::try
 101: rayon_core::registry::in_worker
 102: rayon::iter::plumbing::bridge_producer_consumer::helper
 103: std::panicking::try
 104: rayon_core::registry::in_worker
 105: rayon::iter::plumbing::bridge_producer_consumer::helper
 106: std::panicking::try
 107: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
 108: std::panicking::try
 109: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 110: rayon_core::registry::WorkerThread::wait_until_cold
 111: rayon_core::registry::in_worker
 112: rayon::iter::plumbing::bridge_producer_consumer::helper
 113: std::panicking::try
 114: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 115: rayon_core::registry::WorkerThread::wait_until_cold
 116: rayon_core::registry::in_worker
 117: rayon::iter::plumbing::bridge_producer_consumer::helper
 118: rayon_core::job::StackJob<L,F,R>::run_inline
 119: rayon_core::registry::in_worker
 120: rayon::iter::plumbing::bridge_producer_consumer::helper
 121: rayon_core::job::StackJob<L,F,R>::run_inline
 122: rayon_core::registry::in_worker
 123: rayon::iter::plumbing::bridge_producer_consumer::helper
 124: std::panicking::try
 125: rayon_core::registry::in_worker
 126: rayon::iter::plumbing::bridge_producer_consumer::helper
 127: <rayon::iter::plumbing::bridge::Callback<C> as rayon::iter::plumbing::ProducerCallback<I>>::callback
 128: rayon::iter::collect::special_extend
 129: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend
 130: merkletree::store::Store::build
 131: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::build_partial_tree
 132: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
 133: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
 134: <storage_proofs_core::merkle::tree::MerkleTreeWrapper<H,S,U,V,W> as storage_proofs_core::merkle::tree::MerkleTreeTrait>::gen_cached_proof
 135: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
 136: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
 137: rayon::iter::plumbing::Producer::fold_with
 138: rayon::iter::plumbing::bridge_producer_consumer::helper
 139: std::panicking::try
 140: rayon_core::registry::in_worker
 141: rayon::iter::plumbing::bridge_producer_consumer::helper
 142: std::panicking::try
 143: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 144: rayon_core::registry::WorkerThread::wait_until_cold
 145: rayon_core::registry::in_worker
 146: rayon::iter::plumbing::bridge_producer_consumer::helper
 147: rayon_core::job::StackJob<L,F,R>::run_inline
 148: rayon_core::registry::in_worker
 149: rayon::iter::plumbing::bridge_producer_consumer::helper
 150: std::panicking::try
 151: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 152: rayon_core::registry::WorkerThread::wait_until_cold
 153: rayon_core::registry::in_worker
 154: rayon::iter::plumbing::bridge_producer_consumer::helper
 155: std::panicking::try
 156: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
 157: std::panicking::try
 158: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 159: rayon_core::registry::WorkerThread::wait_until_cold
 160: rayon_core::registry::in_worker
 161: rayon::iter::plumbing::bridge_producer_consumer::helper
 162: std::panicking::try
 163: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 164: rayon_core::registry::WorkerThread::wait_until_cold
 165: rayon_core::registry::in_worker
 166: rayon::iter::plumbing::bridge_producer_consumer::helper
 167: std::panicking::try
 168: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 169: rayon_core::registry::WorkerThread::wait_until_cold
 170: rayon_core::registry::in_worker
 171: rayon::iter::plumbing::bridge_producer_consumer::helper
 172: rayon_core::job::StackJob<L,F,R>::run_inline
 173: rayon_core::registry::in_worker
 174: rayon::iter::plumbing::bridge_producer_consumer::helper
 175: std::panicking::try
 176: rayon_core::registry::in_worker
 177: rayon::iter::plumbing::bridge_producer_consumer::helper
 178: <rayon::iter::plumbing::bridge::Callback<C> as rayon::iter::plumbing::ProducerCallback<I>>::callback
 179: rayon::iter::collect::special_extend
 180: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend
 181: merkletree::store::Store::build
 182: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::build_partial_tree
 183: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
 184: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
 185: <storage_proofs_core::merkle::tree::MerkleTreeWrapper<H,S,U,V,W> as storage_proofs_core::merkle::tree::MerkleTreeTrait>::gen_cached_proof
 186: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
 187: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
 188: rayon::iter::plumbing::Producer::fold_with
 189: rayon::iter::plumbing::bridge_producer_consumer::helper
 190: std::panicking::try
 191: rayon_core::registry::in_worker
 192: rayon::iter::plumbing::bridge_producer_consumer::helper
 193: std::panicking::try
 194: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 195: rayon_core::registry::WorkerThread::wait_until_cold
 196: rayon_core::registry::in_worker
 197: rayon::iter::plumbing::bridge_producer_consumer::helper
 198: std::panicking::try
 199: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 200: rayon_core::registry::WorkerThread::wait_until_cold
 201: rayon_core::registry::in_worker
 202: rayon::iter::plumbing::bridge_producer_consumer::helper
 203: std::panicking::try
 204: rayon_core::registry::in_worker
 205: rayon::iter::plumbing::bridge_producer_consumer::helper
 206: std::panicking::try
 207: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 208: rayon_core::registry::WorkerThread::wait_until_cold
 209: rayon_core::registry::in_worker
 210: rayon::iter::plumbing::bridge_producer_consumer::helper
 211: std::panicking::try
 212: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 213: rayon_core::registry::WorkerThread::wait_until_cold
 214: rayon_core::registry::in_worker
 215: rayon::iter::plumbing::bridge_producer_consumer::helper
 216: std::panicking::try
 217: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 218: rayon_core::registry::WorkerThread::wait_until_cold
 219: rayon_core::registry::in_worker
 220: rayon::iter::plumbing::bridge_producer_consumer::helper
 221: rayon_core::job::StackJob<L,F,R>::run_inline
 222: rayon_core::registry::in_worker
 223: rayon::iter::plumbing::bridge_producer_consumer::helper
 224: rayon_core::job::StackJob<L,F,R>::run_inline
 225: rayon_core::registry::in_worker
 226: rayon::iter::plumbing::bridge_producer_consumer::helper
 227: rayon_core::job::StackJob<L,F,R>::run_inline
 228: rayon_core::registry::in_worker
 229: rayon::iter::plumbing::bridge_producer_consumer::helper
 230: rayon_core::job::StackJob<L,F,R>::run_inline
 231: rayon_core::registry::in_worker
 232: rayon::iter::plumbing::bridge_producer_consumer::helper
 233: <rayon::iter::plumbing::bridge::Callback<C> as rayon::iter::plumbing::ProducerCallback<I>>::callback
 234: rayon::iter::collect::special_extend
 235: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend
 236: merkletree::store::Store::build
 237: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::build_partial_tree
 238: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
 239: merkletree::merkle::MerkleTree<E,A,S,BaseTreeArity,SubTreeArity,TopTreeArity>::gen_cached_proof
 240: <storage_proofs_core::merkle::tree::MerkleTreeWrapper<H,S,U,V,W> as storage_proofs_core::merkle::tree::MerkleTreeTrait>::gen_cached_proof
 241: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
 242: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
 243: rayon::iter::plumbing::Producer::fold_with
 244: rayon::iter::plumbing::bridge_producer_consumer::helper
 245: std::panicking::try
 246: rayon_core::registry::in_worker
 247: rayon::iter::plumbing::bridge_producer_consumer::helper
 248: std::panicking::try
 249: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
 250: rayon_core::registry::WorkerThread::wait_until_cold
 251: rayon_core::registry::ThreadBuilder::run
 252: std::sys_common::backtrace::__rust_begin_short_backtrace
 253: core::ops::function::FnOnce::call_once{{vtable.shim}}
 254: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/cb121987158d69bb894ba1bcc21dc45d1e0a488f/library/alloc/src/boxed.rs:1872:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/cb121987158d69bb894ba1bcc21dc45d1e0a488f/library/alloc/src/boxed.rs:1872:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/cb121987158d69bb894ba1bcc21dc45d1e0a488f/library/std/src/sys/unix/thread.rs:108:17
 255: <unknown>
 256: <unknown>)

It also looks like you are running Lotus with custom code or have made adjustment hence the .dirty
What changes has been made? If it's mistake, please upgrade Lotus to latest stable release v1.18.0.
- I installed the lotus version as v1.18.0 and ran Windows Post, but it did not run.
- Daemon: 1.18.0+mainnet+git.bd10bdf99+api1.5.0

Local: lotus version 1.18.0+mainnet+git.bd10bdf99
Daemon:  1.18.0+mainnet+git.bd10bdf99+api1.5.0
Local: lotus-miner version 1.18.0+mainnet+git.bd10bdf99

lotus-miner proving deadlines result

 Miner: f01393827
deadline  partitions  sectors (faults)  proven partitions
0         1           2298 (2298)       0
1         1           2346 (2346)       0
2         1           2348 (2348)       0
3         1           2349 (2349)       0
4         1           2349 (2349)       0
5         1           2349 (2349)       0
6         1           2349 (2349)       0
7         1           2346 (2346)       0
8         1           2349 (2349)       0
9         1           2349 (2349)       0
10        1           2349 (2349)       0
11        1           2349 (2349)       0
12        1           2341 (2341)       0  (current)
13        1           1491 (1491)       0
14        1           57 (57)           0
15        1           21 (21)           0
16        0           0 (0)             0

Even if you upgrade the version and run lotus, the result is the same.

0 replies

Reiers · 2022-11-19T11:45:57Z

Reiers
Nov 19, 2022

The backup is a bit old, but not old enough to fault all sectors.
Are you 100% sure, that miner has all permissions rwx to the mounts?

Looking at the first, error keystore not found
Did you update the envvars after the re-init ?

export MINER_API_INFO=<newtoken>:ip
export FULLNODE_API_INFO=<newtoken>:ip

When running:
lotus-miner actor control list does the command output the right control addresses, such as owner and worker.

Are you able to:
lotus-miner actor withdraw
Just checking that miner has access to your node.

I also noticed that there are no DeclareFaultsRecovered messages sent out before the wdPosts.
Could also try with a different GPU? Just to rule out that it is not hardware related ?

0 replies

MMNNMMNM · 2022-11-20T06:42:22Z

MMNNMMNM
Nov 20, 2022
Author

The backup is a bit old, but not old enough to fault all sectors.
Are you 100% sure, that miner has all permissions rwx to the mounts?
- Miner has enough rights to mount.
Looking at the first, error keystore not found
Did you update the envvars after the re-init ?

export MINER_API_INFO=:ip
export FULLNODE_API_INFO=:ip
When running:
lotus-miner actor control list does the command output the right control addresses, such as owner and worker.

Yes I did an envvar update.
Are you able to:
lotus-miner actor withdraw
Just checking that miner has access to your node.
- Coins can be withdrawn from miner.

I also noticed that there are no DeclareFaultsRecovered messages sent out before the wdPosts.
Could also try with a different GPU? Just to rule out that it is not hardware related ?

I confirmed that the GPU is running, but the window post is not running.

0 replies

MMNNMMNM · 2022-11-21T07:47:08Z

MMNNMMNM
Nov 21, 2022
Author

We tried sealing 3 sectors on the troubled miner.
As a result, the three sectors that were sealed today successfully window-posted.
I don't know what the hell is wrong with the rest of the sectors where the window-post has failed. I desperately need help.

The image below is a window-post computed image of the deadline with the 3 sectors we recently sealed.

0 replies

Reiers · 2022-11-21T11:28:49Z

Reiers
Nov 21, 2022

What was the command you use to re-init the miner?

You changed owner address a while back.

Sectors:
        Total: 0

Workers: Seal(1) WdPoSt(0) WinPoSt(0)

Storage Deals: 0, 0 B

Retrieval Deals (complete): 0, 0 B

Looking at this, the sectors doesn't even register and all the deals are also gone.

Did you use the right addresses to re-init?
How often do you take backups, and do you change the name of the backup each time so the file doesn't corrupt?

0 replies

Reiers · 2022-11-21T11:35:14Z

Reiers
Nov 21, 2022

We tried sealing 3 sectors on the troubled miner.
As a result, the three sectors that were sealed today successfully window-posted.

There's no evidence of that you sealed 3 sectors today, there are no PC2 or C2 since December 2021.

Not trying to call you out here, just want to figure out what the issue is.

So are you running multiple miner's, and made a mistake?
Can you share the log status of the newly created sectors? Would be interesting to see which address, minerID owns those new sectors.

0 replies

MMNNMMNM · 2022-11-21T12:11:32Z

MMNNMMNM
Nov 21, 2022
Author

What was the command you use to re-init the miner?
You changed owner address a while back.

  Sectors:
            Total: 0
  Workers: Seal(1) WdPoSt(0) WinPoSt(0)
  Storage Deals: 0, 0 B
  Retrieval Deals (complete): 0, 0 B

Looking at this, the sectors doesn't even register and all the deals are also gone.

○ We used the following command in our minor init
lotus-miner init restore /path/backup.cbor

Did you use the right addresses to re-init?
○ When initializing, I obviously used the correct address.
How often do you take backups, and do you change the name of the backup each time so the file doesn't corrupt?
○ The backup is the most recent 2022-09-27.
Cbor file has never been renamed.

0 replies

Reiers · 2022-11-21T12:36:41Z

Reiers
Nov 21, 2022

Thanks for the information !

Apart from adding the flags to the restore cmd, lotus-miner init restore /path/backup.cbor

   --config value          config file (config.toml)
   --storage-config value  storage paths config (storage.json)

I don't see anything wrong, because I guess you copied the both files after the init anyway.

○ When initializing, I obviously used the correct address.

I meant the correct backup file. It's was early in the day here and mistook the cmd for init / restore, exp:
lotus-miner init --owner=<address> --worker=<address>
Sorry about that !

○ The backup is the most recent 2022-09-27.
Cbor file has never been renamed.

So every backup is been named backup.cbor ? Do you delete the backup before making a new one in that path? /path/backup.cbor
Just want to be sure, that the backup files are separated so there's no corruption in the file.

Also missing a answer for:

Can you share the log status of the newly created sectors? Would be interesting to see which address, minerID owns those new sectors?

0 replies

MMNNMMNM · 2022-11-21T12:37:39Z

MMNNMMNM
Nov 21, 2022
Author

There's no evidence of that you sealed 3 sectors today, there are no PC2 or C2 since December 2021.
Not trying to call you out here, just want to figure out what the issue is.

So are you running multiple miner's, and made a mistake?
Can you share the log status of the newly created sectors? Would be interesting to see which address, minerID owns those new sectors.

○ There is a miner that has the same symptoms as the miner above.
For the f01625485 miner, 3 sectors were sealed, and the 3 recently sealed sectors succeeded until window-post.

  SectorID:   28272
   Status:      Proving
   CIDcommD:   baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
   CIDcommR:   bagboea4b5abcb6gnrvdqjrblbtuknwyntojqnieidukvj2sn6hokocddbz4apck7
   Ticket:      530725072d2fe140e49bda2d81e1c752a65c714762637cbd551edca81ffbf617
   TicketH:   2342350
   Seed:      c434274f22da2680fca8f33f5d5122735a0faa48ac9dfc9273435dc4a62344da
   SeedH:      2346280
   Precommit:   bafy2bzacebjw5pyraq56xmqkfymedilleokwe7xotbaqvvlty5vjcqwe42occ
   Commit:      bafy2bzacebmjzve4ehqylm5mvsjrg7w3pchwjqwuwb6nlz5yet7jiwqufg5p2
   Deals:      [0]
   Retries:   0
--------
Event Log:
0.   2022-05-25 13:30:59 +0900 KST:   [event;sealing.SectorStart]   {"User":{"ID":28272,"SectorType":8}}
1.   2022-05-25 13:30:59 +0900 KST:   [event;sealing.SectorAddPiece]   {"User":{}}
2.   2022-06-17 11:21:52 +0900 KST:   [event;sealing.SectorPieceAdded]   {"User":{"NewPieces":null}}
3.   2022-06-17 11:22:07 +0900 KST:   [event;sealing.SectorStartPacking]   {"User":{}}
4.   2022-11-14 17:33:49 +0900 KST:   [event;sealing.SectorPacked]   {"User":{"FillerPieces":[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]}}
5.   2022-11-14 17:33:50 +0900 KST:   [event;sealing.SectorTicket]   {"User":{"TicketValue":"0PeI1DDobmeXVO2x7znIleca2l7u8B+zhMaJc8FmlM8=","TicketEpoch":2336047}}
6.   2022-11-14 19:55:36 +0900 KST:   [event;sealing.SectorSealPreCommit1Failed]   {"User":{}}
  seal pre commit(1) failed: storage call error 101: worker [name: main] restarted
7.   2022-11-14 19:56:36 +0900 KST:   [event;sealing.SectorRetrySealPreCommit1]   {"User":{}}
8.   2022-11-15 02:18:34 +0900 KST:   [event;sealing.SectorSealPreCommit1Failed]   {"User":{}}
  seal pre commit(1) failed: storage call error 101: worker [name: main] restarted
9.   2022-11-15 02:19:34 +0900 KST:   [event;sealing.SectorRetrySealPreCommit1]   {"User":{}}
10.   2022-11-15 21:56:55 +0900 KST:   [event;sealing.SectorOldTicket]   {"User":{}}
11.   2022-11-15 21:56:55 +0900 KST:   [event;sealing.SectorTicket]   {"User":{"TicketValue":"npZyJvI73W40bw67JQkvbhJkMyGvH6/OCaYBl2LDtl0=","TicketEpoch":2339453}}
12.   2022-11-16 19:38:41 +0900 KST:   [event;sealing.SectorPreCommit1]   {"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJucFp5SnZJNzNXNDBidzY3SlFrdmJoSmtNeUd2SDYvT0NhWUJsMkxEdGwwPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvaG9tZS9pay8ubG90dXNtaW5lci9jYWNoZS9zLXQwMTYyNTQ4NS0yODI3MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MjE0NzQ4MzY0N30sImxhYmVscyI6eyJTdGFja2VkRHJnMzJHaUJWMSI6eyJfaCI6bnVsbCwibGFiZWxzIjpbeyJpZCI6ImxheWVyLTEiLCJwYXRoIjoiL2hvbWUvaWsvLmxvdHVzbWluZXIvY2FjaGUvcy10MDE2MjU0ODUtMjgyNzIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci0yIiwicGF0aCI6Ii9ob21lL2lrLy5sb3R1c21pbmVyL2NhY2hlL3MtdDAxNjI1NDg1LTI4MjcyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvaG9tZS9pay8ubG90dXNtaW5lci9jYWNoZS9zLXQwMTYyNTQ4NS0yODI3MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTQiLCJwYXRoIjoiL2hvbWUvaWsvLmxvdHVzbWluZXIvY2FjaGUvcy10MDE2MjU0ODUtMjgyNzIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci01IiwicGF0aCI6Ii9ob21lL2lrLy5sb3R1c21pbmVyL2NhY2hlL3MtdDAxNjI1NDg1LTI4MjcyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvaG9tZS9pay8ubG90dXNtaW5lci9jYWNoZS9zLXQwMTYyNTQ4NS0yODI3MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTciLCJwYXRoIjoiL2hvbWUvaWsvLmxvdHVzbWluZXIvY2FjaGUvcy10MDE2MjU0ODUtMjgyNzIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci04IiwicGF0aCI6Ii9ob21lL2lrLy5sb3R1c21pbmVyL2NhY2hlL3MtdDAxNjI1NDg1LTI4MjcyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvaG9tZS9pay8ubG90dXNtaW5lci9jYWNoZS9zLXQwMTYyNTQ4NS0yODI3MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTEwIiwicGF0aCI6Ii9ob21lL2lrLy5sb3R1c21pbmVyL2NhY2hlL3MtdDAxNjI1NDg1LTI4MjcyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTEiLCJwYXRoIjoiL2hvbWUvaWsvLmxvdHVzbWluZXIvY2FjaGUvcy10MDE2MjU0ODUtMjgyNzIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
13.   2022-11-16 22:04:06 +0900 KST:   [event;sealing.SectorPreCommit2]   {"User":{"Sealed":{"/":"bagboea4b5abcarfudjaa4e5k5l66lc6soholt76i2nlqfggh6byuj5ljsieekwzp"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
14.   2022-11-16 22:04:06 +0900 KST:   [event;sealing.SectorPreCommitBatch]   {"User":{}}
15.   2022-11-16 22:04:06 +0900 KST:   [event;sealing.SectorSealPreCommit1Failed]   {"User":{}}
  ticket expired: ticket expired: seal height: 2340353, head: 2343248
16.   2022-11-16 22:05:06 +0900 KST:   [event;sealing.SectorRetrySealPreCommit1]   {"User":{}}
17.   2022-11-16 22:05:06 +0900 KST:   [event;sealing.SectorOldTicket]   {"User":{}}
18.   2022-11-16 22:05:06 +0900 KST:   [event;sealing.SectorTicket]   {"User":{"TicketValue":"UwclBy0v4UDkm9otgeHHUqZccUdiY3y9VR7cqB/79hc=","TicketEpoch":2342350}}
19.   2022-11-17 19:29:35 +0900 KST:   [event;sealing.SectorPreCommit1]   {"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJVd2NsQnkwdjRVRGttOW90Z2VISFVxWmNjVWRpWTN5OVZSN2NxQi83OWhjPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvaG9tZS9pay8ubG90dXNtaW5lci9jYWNoZS9zLXQwMTYyNTQ4NS0yODI3MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MjE0NzQ4MzY0N30sImxhYmVscyI6eyJTdGFja2VkRHJnMzJHaUJWMSI6eyJfaCI6bnVsbCwibGFiZWxzIjpbeyJpZCI6ImxheWVyLTEiLCJwYXRoIjoiL2hvbWUvaWsvLmxvdHVzbWluZXIvY2FjaGUvcy10MDE2MjU0ODUtMjgyNzIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci0yIiwicGF0aCI6Ii9ob21lL2lrLy5sb3R1c21pbmVyL2NhY2hlL3MtdDAxNjI1NDg1LTI4MjcyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvaG9tZS9pay8ubG90dXNtaW5lci9jYWNoZS9zLXQwMTYyNTQ4NS0yODI3MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTQiLCJwYXRoIjoiL2hvbWUvaWsvLmxvdHVzbWluZXIvY2FjaGUvcy10MDE2MjU0ODUtMjgyNzIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci01IiwicGF0aCI6Ii9ob21lL2lrLy5sb3R1c21pbmVyL2NhY2hlL3MtdDAxNjI1NDg1LTI4MjcyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvaG9tZS9pay8ubG90dXNtaW5lci9jYWNoZS9zLXQwMTYyNTQ4NS0yODI3MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTciLCJwYXRoIjoiL2hvbWUvaWsvLmxvdHVzbWluZXIvY2FjaGUvcy10MDE2MjU0ODUtMjgyNzIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci04IiwicGF0aCI6Ii9ob21lL2lrLy5sb3R1c21pbmVyL2NhY2hlL3MtdDAxNjI1NDg1LTI4MjcyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvaG9tZS9pay8ubG90dXNtaW5lci9jYWNoZS9zLXQwMTYyNTQ4NS0yODI3MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTEwIiwicGF0aCI6Ii9ob21lL2lrLy5sb3R1c21pbmVyL2NhY2hlL3MtdDAxNjI1NDg1LTI4MjcyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTEiLCJwYXRoIjoiL2hvbWUvaWsvLmxvdHVzbWluZXIvY2FjaGUvcy10MDE2MjU0ODUtMjgyNzIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
20.   2022-11-17 22:04:10 +0900 KST:   [event;sealing.SectorPreCommit2]   {"User":{"Sealed":{"/":"bagboea4b5abcb6gnrvdqjrblbtuknwyntojqnieidukvj2sn6hokocddbz4apck7"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
21.   2022-11-17 22:04:10 +0900 KST:   [event;sealing.SectorPreCommitBatch]   {"User":{}}
22.   2022-11-17 22:04:18 +0900 KST:   [event;sealing.SectorPreCommitBatchSent]   {"User":{"Message":{"/":"bafy2bzacebjw5pyraq56xmqkfymedilleokwe7xotbaqvvlty5vjcqwe42occ"}}}
23.   2022-11-17 22:08:00 +0900 KST:   [event;sealing.SectorPreCommitLanded]   {"User":{"TipSet":[{"/":"bafy2bzaceanfiykhkis23fx35pjxb5ah5kaxnjee75ilzkutxpceeux2nwrgk"},{"/":"bafy2bzacedoinzfe5vs43lcw7rsahqznpur6h7ky6h73ip2pnb7ybik7fxxxa"},{"/":"bafy2bzaced24nuqtyszcv7klfb32wmluhphcr4qpyuuhvxicj75cvdtavoiiy"},{"/":"bafy2bzacedc6ean4oroxoqaluyw5xefk74j6zqgqiax3gum2csxps7lqkc4iw"}]}}
24.   2022-11-17 23:22:59 +0900 KST:   [event;sealing.SectorSeedReady]   {"User":{"SeedValue":"xDQnTyLaJoD8qPM/XVEic1oPqkisnfySc0NdxKYjRNo=","SeedEpoch":2346280}}
25.   2022-11-18 01:00:35 +0900 KST:   [event;sealing.SectorCommitted]   {"User":{"Proof":"lbxKZd2acCGgQXXxi9xKywl1DiIUdBT1sbKqlI0yRmj6O24FN1OidOXOAUSAN3GJtOtCXORFrYuN0M4nFaUFCNOqAHlBlbI1NLGAMwP5eeVAygG1Xav3toOpYfPcQDJFFiqUpk5GLuYSeMuZSJ5I25VPeSDm9F7zkJyTQh6ugqQqeVGGeQqTEncjKqluKl0btlQcoNKYVE4D2uyGmmSP/+hMWwT++xgREkersK672ta7Tp7Opz5KWKTYxZL2kXyHt+haLqiWVVSSj/A3ne4RLUxBA+cRxgOKNmpFl4FWCwSAmRCYItW/S0jSFL6BeUGgrhymYC25xBLyU6PD41Wxq+b27zQ7d4cJJPaczrXwfF9PQKwlRbWMOp2lc0z0f87eBFsbXmvQCguw22D5dL/OQHOW2wOsdh1r951RTClUABLVjIJG31UhF+sdfaphNWbkhAcMzaqsoUhj4KgSE4TZhQm+9voEukOf0XABX1IZ9Vfs1Qa+MZ/BxOhdquAbzME5kDUHET578qHXAuCEt9JiIEEitt8bY3HmVRhTTnIbhaD8kpR0ByKoiyJ6s01UV8Xohj2za2XZgMg+Rs9igU2D6w5jedXWP1kxsKlWsXm1V13yNZIVfsr22RJX05wiRErmE1ymrnvpg1SB67WAlz3pLHyrgIkD7b/ApwzRp4q0qx4KedV9fMjyMLxuWoCOH+qmiUExXYg+iju/SuEJwZrcSWJOE7XFuoWS92log94jznB8kd4I1z9Y0btfVPDRa51Hl7Qte6baLkOad62vGnksNGM9YB4HwVLDb5HEZRYimwWQSs/eMbHKkiLMuaTCnr+JsmeQwxQFTQEwKGiQnDp1ysg8mRTNL2gD1e+SidpfqPU8aa4mFBsJuYeNWFwAwqWjGQ059tqqOQho8DxoccinUk/irGBGw9V82CZq3pbSayH2J8AMq1B+6zJA5gft0wpmiE4tkeRN7uejZq5PePC7y/DVcOOee06LSiDeIUbUN1o2nCMoqFSQq9KjLMvQSm9Xht9w+e1OdOmSc1KnL1/U2WYvObJngSrPJ9EmVMbqmrBvpif+31d+AcoBXdaeFiydqIR620F2v9Buqk7tE42VrmpGAz4dC91tqVZ2qnIIbjqiff8SvXpctxVJyBYwtbMAC/xQ2aQoGofTI0PfKNgGijmVWmaLuuby76CdTaUvINUISI8WcTIKqvez36UQEJvxhPh+Nu/04UrAMc+cDNM+pxSBwCZTUy7N6rGg0jKJzLhaHB/ZvF/D40bs9Yy0plEOlwgdDk6lzAm9/JwuIs4sq3JXQKALqVf0/a94TNx4Du+PnzVLhCVjh9iEeYS1LbgGgvLl+aMppF1DzWvvA5pNXlie8JPytEy61KslQtSa5Ws1Aaoyj0OwMlzX+UGrmACcFb7eeX7FXUqJgyGFgM53t8mm6gadRs+rKKujNYjpvRTwwVgFG7rFR2n4iR8qZHiKhIDezcIxUrHNVZwZBV0c+9vkn8wCG1EBZEhvCIn1rDREPr8LCAx4AAmAqaJ+9Y7Wka9+CWDESoT+KcSs+8MZiH9k7j6M9zKD6YJq3T9Fj6PoF1MV/4Yp5bOZpdULJHNRk+dsLv37Mauostim0PaF0TXvdYB9oFPFT4tWGfScLkXIoaaMMF+Cd+SGoyrGVtvMCQ514HmXoPAAXD6qSxDXx0N3BXj9N/USaP+GSk9UWTSoVjiJPfI+30m23HQr5CPwlmw4EMfD4se/sHjHWDFtBO2Qo/abwyWlX09p3bSftfQbZC+/ST8IgEUuBKiu61LjgWuxIyDvtkXdK3eOv/TeFABZpxrpHyZ+8ub4xzJIxGS7jehD9bZj+NnQ0W08F3d8iqXPXCD9Lc2qCPAIdl4L9LSdUfk9bMl2S0Td4DTkMDLB8ByHmcqbHvuC4rfKm/opDXfS26l9iLm58EeO1yJOQow94bLVXdiFe5D3rtHL1nuC0JtjwW91Z5Kpa+9171wxmeTYPNQpYg4mJ71Xvzo/f6yWGILmhlMnVOvCNBS0YlfkPtgOfUVLjuGkhcmq4jtjtaHfQP413J9YKz56dqSjAjW/kkt1xe1uP5vKLEKeuMUpnwtgcKWj3FBm7oR2fD04kibcq35dhN3EM/dqwMLtKP97Q4jaMygV4jM3ixLeqLQGK+RUuknTlV7qTtS+OUOAEB83nnBIBZy4c0YtdKW4Y09K6XFlXDBIuZUe054yhy+S6SNm1rgWrxabDeHvCvTRlhLkD5ZOG1AmoaFcAu+2H5z0lYPauudRcQtcaWAGaZi/0QAtC6yguAiRbzcuca0KpxNcpq8k0k4GU+ypiGpefepnFI9YcPDavl28JpxkKaZMxCOKC3R5gdwKCtG3PN3cuXMpWFCbm2RmjtZJTleC22GmgnekVhoqFBNlVueBMolIjy69D5WE9qWsQFvAVSWZCWRE0vBQy+piE+qAyzu128zDhdHpc1jlcXn/tNXrnODTQ5uOrqj5lPQA9mB+3zjetcCusuDRXiXeM9Rcwyy4YZOE6PpBhmSwz1YBIk6KEaxU+xxXMA4dMnPiNKnzAF0j"}}
26.   2022-11-18 01:00:35 +0900 KST:   [event;sealing.SectorSubmitCommitAggregate]   {"User":{}}
27.   2022-11-20 03:38:09 +0900 KST:   [event;sealing.SectorCommitAggregateSent]   {"User":{"Message":{"/":"bafy2bzacebmjzve4ehqylm5mvsjrg7w3pchwjqwuwb6nlz5yet7jiwqufg5p2"}}}
28.   2022-11-20 03:41:29 +0900 KST:   [event;sealing.SectorProving]   {"User":{}}
29.   2022-11-20 03:41:38 +0900 KST:   [event;sealing.SectorFinalized]   {"User":{}}

0 replies

Reiers · 2022-11-21T12:44:25Z

Reiers
Nov 21, 2022

Okay - so there are 2 minerIDs.

f01393827 and f01625485

has the same symptoms

Did you re-init this miner as well? In the same way, or did it just stop showing windowPost?

Your first report was about f01393827 can you seal a sector on that one as well?

0 replies

MMNNMMNM · 2022-11-21T12:50:41Z

MMNNMMNM
Nov 21, 2022
Author

Okay - so there are 2 minerIDs.

f01393827 and f01625485

has the same symptoms

Did you re-init this miner as well? In the same way, or did it just stop showing windowPost?
Your first report was about f01393827 can you seal a sector on that one as well?

○ Both miners did the same init.
f01625485 has seen it seal, but f01393827 has not yet.

0 replies

Reiers · 2022-11-21T13:13:28Z

Reiers
Nov 21, 2022

I need to summarize the events here, because I having a hard time understanding the events that led this issue.

You stated that the first "Sector Fault happened" on 2022-11-13T09:15:29, on the same day you downloaded and snapshot - re-sync'ed, but when you started lotus-miner run the output was:
"keystore not found"

Then your next step was then to re-init the miner from a backup.

Both miners did the same init.

Now it's 2 minerID's that went down on the same day. Both so broken that they needed to be re-init'ed.

I need to asked again, since you didn't answer:
So every backup is been named backup.cbor ? Do you delete the backup before making a new one in that path? /path/backup.cbor
Just want to be sure, that the backup files are separated so there's no corruption in the file.

What actually happened that day?

0 replies

MMNNMMNM · 2022-11-21T13:23:06Z

MMNNMMNM
Nov 21, 2022
Author

I need to asked again, since you didn't answer:
So every backup is been named backup.cbor ? Do you delete the backup before making a new one in that path? /path/backup.cbor
Just want to be sure, that the backup files are separated so there's no corruption in the file.
All backup files are named backup.cbor.
- All backup files are named backup.cbor.
  When receiving a backup file, the cbor file is not deleted because it is classified by date.
  If I haven't answered your other questions, please ask for more.

0 replies

Reiers · 2022-11-21T14:05:00Z

Reiers
Nov 21, 2022

When receiving a backup file, the cbor file is not deleted because it is classified by date.

Okay, so every time you have taken a backup - it has overwritten the backup.cbor ?
That may have caused the backup file to corrupt.

Trying to think of way's for you to recover.

The old miner repo/path/filed from before the re-init, is it still intact or is it deleted?

0 replies

MMNNMMNM · 2022-11-21T16:42:55Z

MMNNMMNM
Nov 21, 2022
Author

The old miner repo/path/filed from before the re-init, is it still intact or is it deleted?
- Files in lotusminer's datastore/metadata were deleted before init.

0 replies

MMNNMMNM · 2022-11-23T12:14:52Z

MMNNMMNM
Nov 23, 2022
Author

Our miner's situation

The miner has storage with sealed sectors and the each storage list has cache, sealed, unsealed sector data.

   lotus-miner storage find 0
   In 9649cf77-c806-4ca6-ae15-50390dfa288b (Sealed, Cache)
        Sealing: false; Storage: true
        Local (/root/.lotusminer1)
        URL: http://127.0.0.1:2345/remote/sealed/s-t01393827-0

We has wallet data. (imported wallet data, key file)
The lotus daemon and lotus-miner are running on a single server.
(No WdPoSt or WinPoSt worker.)

@Reiers As you said that the backup file may be damaged then please let us know is there any way to solve the problem with only the above data without the cbor file?
Do you know how to restore the miner without backup cbor file?

0 replies

rjan90 · 2022-11-23T13:13:41Z

rjan90
Nov 23, 2022
Maintainer

Hey @MMNNMMNM!

As long as you have the sectors and the owner-key for the Lotus-Miner, you should be able to re-initialize with lotus-miner init --actor. That said!, re-initializing with this method will not get your metadata-back, and sealing new sectors, doing deals, etc, will most likely be not possible on these MinerIDs.

It should however make you able to prove your sectors and do winningPoSt until your sectors expire.

1 reply

lbj2004032 Mar 19, 2023

Hey @MMNNMMNM!

As long as you have the sectors and the owner-key for the Lotus-Miner, you should be able to re-initialize with lotus-miner init --actor. That said!, re-initializing with this method will not get your metadata-back, and sealing new sectors, doing deals, etc, will most likely be not possible on these MinerIDs.

It should however make you able to prove your sectors and do winningPoSt until your sectors expire.

Windowpost failed,

The $lotus miner providing deadlines command displays a number of faults that match, but cannot execute windowpost correctly

MMNNMMNM · 2022-11-24T08:35:48Z

MMNNMMNM
Nov 24, 2022
Author

miner condition
export LOTUS_MINER_PATH=.lotusminer0

First case

used : lotus-miner init --actor f01393827 --no-local-storage
result : ERROR: repo at '.lotusminer0' is already initialized

Second case

export LOTUS_MINER_PATH=/.lotusminer3

used:
lotus-miner init --actor f01393827 --no-local-storage
cp .lotusminer0/storage.json .lotusminer3/storage.json
lotus-miner run
```
 `lotus-miner proving compute window-post 8`
```
result :
Took 968.587533ms []

We want to use the ‘--actor’ as you recommended for initializing the miner but we get the same result that we got before using the ‘init --actor’.
@Reiers @rjan90 ls there any missing command or the way we did not try?

2 replies

rjan90 Nov 24, 2022
Maintainer

Can you check what you get with the lotus-miner proving check --slow 8 ?

MMNNMMNM Nov 25, 2022
Author

This result is obtained with check --slow 8 .

result :
Took 968.587533ms []

Reiers · 2022-11-24T14:00:11Z

Reiers
Nov 24, 2022

Hi @MMNNMMNM

As we can see that issue is local, and lies elsewhere.
Moving this issue to Lotus Discussions for help and troubleshooting.

0 replies

MMNNMMNM · 2022-11-27T12:04:27Z

MMNNMMNM
Nov 27, 2022
Author

Our storage miner had .lotusminer ~ .lotusminer4, and among them, LOTUS_MINER_PATH was hanging on .lotusminer.
While we were running the miner, all the files in /.lotusminer/datastore/metadata disappeared.
When I first tried to restore the .lotusminer directory, it didn't work.
So after creating the .lotusminer0 directory, I restored it using the cbor file and changed the LOTUS_MINER_PATH to .lotusminer0.
The command used when restoring the cbor file is as follows.

lotus-miner init --actor f01393827 --no-local-storage
cp .lotusminer/storage.json .lotusminer0/storage.json
cp .lotusminer/config.toml .lotusminer0/config.toml
lotus-miner run

After that, miner and daemon are running, but window-post is failing.

We are running Daemon and miner together on one PC.
Is there anything wrong with us restoring the miner?

1 reply

lbj2004032 Mar 19, 2023

Our storage miner had .lotusminer ~ .lotusminer4, and among them, LOTUS_MINER_PATH was hanging on .lotusminer.

While we were running the miner, all the files in /.lotusminer/datastore/metadata disappeared.

When I first tried to restore the .lotusminer directory, it didn't work.
So after creating the .lotusminer0 directory, I restored it using the cbor file and changed the LOTUS_MINER_PATH to .lotusminer0.

The command used when restoring the cbor file is as follows.

lotus-miner init --actor f01393827 --no-local-storage

cp .lotusminer/storage.json .lotusminer0/storage.json

cp .lotusminer/config.toml .lotusminer0/config.toml

lotus-miner run

After that, miner and daemon are running, but window-post is failing.

We are running Daemon and miner together on one PC. Is there anything wrong with us restoring the miner?

Have you solved this problem

MMNNMMNM · 2022-11-28T09:26:51Z

MMNNMMNM
Nov 28, 2022
Author

Below are logs that appear when using the 'lotus-miner init restore --config /path/config.toml --storage-config /path/stroage.json /path/backup.cbor' command.

2022-11-28T18:12:47.122+0900 ERROR sectors storage-sealing/fsm.go:34 unhandled sector error (0): entered TerminateWait with nil TerminateMessage:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).handleTerminateWait
/home/ik/lotus/extern/storage-sealing/states_proving.go:81
2022-11-28T18:12:49.229+0900 ERROR sectors storage-sealing/states_sealing.go:227 handlePreCommit1: StateSectorPreCommitInfo: api error, not proceeding: sectorNumber is allocated, but PreCommit info wasn't found on chain
2022-11-28T18:12:49.326+0900 ERROR sectors storage-sealing/fsm.go:34 unhandled sector error (0): entered TerminateWait with nil TerminateMessage:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).handleTerminateWait
/home/ik/lotus/extern/storage-sealing/states_proving.go:81
2022-11-28T18:12:49.361+0900 DEBUG advmgr sector-storage/sched.go:361 SCHED 0 queued; 2 open windows
2022-11-28T18:12:49.751+0900 ERROR sectors storage-sealing/states_sealing.go:227 handlePreCommit1: StateSectorPreCommitInfo: api error, not proceeding: sectorNumber is allocated, but PreCommit info wasn't found on chain
2022-11-28T18:12:49.752+0900 ERROR sectors storage-sealing/states_sealing.go:227 handlePreCommit1: StateSectorPreCommitInfo: api error, not proceeding: sectorNumber is allocated, but PreCommit info wasn't found on chain
2022-11-28T18:12:49.978+0900 ERROR sectors storage-sealing/fsm.go:34 unhandled sector error (712): entered TerminateWait with nil TerminateMessage:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).handleTerminateWait
/home/ik/lotus/extern/storage-sealing/states_proving.go:81
2022-11-28T18:12:50.198+0900 ERROR sectors storage-sealing/fsm.go:34 unhandled sector error (763): entered TerminateWait with nil TerminateMessage:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).handleTerminateWait
/home/ik/lotus/extern/storage-sealing/states_proving.go:81
2022-11-28T18:12:50.350+0900 ERROR sectors storage-sealing/fsm.go:34 unhandled sector error (791): entered TerminateWait with nil TerminateMessage:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).handleTerminateWait
/home/ik/lotus/extern/storage-sealing/states_proving.go:81
2022-11-28T18:12:50.443+0900 ERROR sectors storage-sealing/fsm.go:34 unhandled sector error (809): entered TerminateWait with nil TerminateMessage:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).handleTerminateWait
/home/ik/lotus/extern/storage-sealing/states_proving.go:81
2022-11-28T18:12:50.463+0900 ERROR sectors storage-sealing/fsm.go:34 unhandled sector error (815): entered TerminateWait with nil TerminateMessage:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).handleTerminateWait
/home/ik/lotus/extern/storage-sealing/states_proving.go:81
2022-11-28T18:12:50.484+0900 ERROR sectors storage-sealing/fsm.go:34 unhandled sector error (820): entered TerminateWait with nil TerminateMessage:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).handleTerminateWait
/home/ik/lotus/extern/storage-sealing/states_proving.go:81
2022-11-28T18:12:50.899+0900 ERROR sectors storage-sealing/fsm.go:34 unhandled sector error (9175): getting precommit info:
github.com/filecoin-project/lotus/extern/storage-sealing.(*Sealing).handleWaitSeed

0 replies

MMNNMMNM · 2022-11-29T05:05:48Z

MMNNMMNM
Nov 29, 2022
Author

@Reiers
I wrote about the problems and phenomena we are experiencing, and I wonder if a solution is being prepared.

0 replies

WindowPost failed after lotus-miner init restore #9724

Checklist

Lotus component

Lotus Version

Describe the Bug

Logging Information

Repo Steps

Replies: 23 comments · 5 replies

MMNNMMNM Nov 19, 2022 Author

MMNNMMNM Nov 20, 2022 Author

MMNNMMNM Nov 21, 2022 Author

MMNNMMNM Nov 21, 2022 Author

MMNNMMNM Nov 21, 2022 Author

MMNNMMNM Nov 21, 2022 Author

MMNNMMNM Nov 21, 2022 Author

MMNNMMNM Nov 21, 2022 Author

MMNNMMNM Nov 23, 2022 Author

Our miner's situation

rjan90 Nov 23, 2022 Maintainer

MMNNMMNM Nov 24, 2022 Author

rjan90 Nov 24, 2022 Maintainer

MMNNMMNM Nov 25, 2022 Author

MMNNMMNM Nov 27, 2022 Author

MMNNMMNM Nov 28, 2022 Author

MMNNMMNM Nov 29, 2022 Author

Replies: 23 comments 5 replies

MMNNMMNM
Nov 19, 2022
Author

MMNNMMNM
Nov 20, 2022
Author

MMNNMMNM
Nov 21, 2022
Author

MMNNMMNM
Nov 21, 2022
Author

MMNNMMNM
Nov 21, 2022
Author

MMNNMMNM
Nov 21, 2022
Author

MMNNMMNM
Nov 21, 2022
Author

MMNNMMNM
Nov 21, 2022
Author

MMNNMMNM
Nov 23, 2022
Author

rjan90
Nov 23, 2022
Maintainer

MMNNMMNM
Nov 24, 2022
Author

rjan90 Nov 24, 2022
Maintainer

MMNNMMNM Nov 25, 2022
Author

MMNNMMNM
Nov 27, 2022
Author

MMNNMMNM
Nov 28, 2022
Author

MMNNMMNM
Nov 29, 2022
Author