Since yesterday my WinPost are giving me issues

[Angelo-gh3990]

Hello,
Since yesterday my WinPost are giving me issues.
The message : https://filfox.info/en/message/bafy2bzacebqpdaiezajene5djqdhriuypsxtkkon62q6pam2sz6pw5wesgyry
Error Message : invalid deadline 0 at epoch 512049, expected 1 (RetCode=16)

Today I got : https://filfox.info/en/message/bafy2bzaceckyjp2kxwjvuh7mccaojqqlzf4dtttn4kilgghs4uq7o2j6p7wis
Method : DeclareFaultsRecovered

Which seemed fine to me, below my logging:

2021-02-19T17:21:24.766 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2021-02-19T17:21:24.911 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0.875)
2021-02-19T17:21:24.911 INFO bellperson::gpu::multiexp > Multiexp: Device 0: GeForce RTX 2070 (Chunk-size: 3964764)
2021-02-19T17:21:24.911 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2021-02-19T17:34:27.732 INFO bellperson::groth16::prover > prover time: 988.092116765s
2021-02-19T17:34:27.867 INFO storage_proofs_core::compound_proof > snark_proof:finish
2021-02-19T17:34:27.867 INFO filecoin_proofs::api::post > generate_window_post:finish
2021-02-19T17:34:27.873 INFO filcrypto::proofs::api > generate_window_post: finish
2021-02-19T17:34:27.874+0100 INFO storageminer storage/wdpost_run.go:595 computing window post {"batch": 0, "elapsed": 1167.405481668}
2021-02-19T17:34:27.892 INFO filcrypto::proofs::api > verify_window_post: start
2021-02-19T17:34:27.895 INFO filecoin_proofs::api::post > verify_window_post:start
2021-02-19T17:34:27.895 INFO filecoin_proofs::caches > trying parameters memory cache for: WINDOW_POST[34359738368]-verifying-key
2021-02-19T17:34:27.895 INFO filecoin_proofs::caches > no params in memory cache for WINDOW_POST[34359738368]-verifying-key
2021-02-19T17:34:27.915 INFO storage_proofs_core::parameter_cache > parameter set identifier for cache: FallbackPoSt::PublicParams{sector_size: 34359738368, challenge_count: 10, sector_count: 2349}
2021-02-19T17:34:27.915 INFO storage_proofs_core::parameter_cache > ensuring that all ancestor directories for: "/var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk" exist
2021-02-19T17:34:27.915 INFO storage_proofs_core::parameter_cache > checking cache_path: "/var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk" for verifying key
2021-02-19T17:34:34.331 INFO storage_proofs_core::parameter_cache > read verifying key from cache "/var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk"
2021-02-19T17:34:38.233 INFO filecoin_proofs::api::post > verify_window_post:finish
2021-02-19T17:34:38.233 INFO filcrypto::proofs::api > verify_window_post: finish
2021-02-19T17:34:41.738+0100 INFO storageminer storage/wdpost_run.go:767 Submitted window post: bafy2bzacealn3r7midsvnquxebemxunmxumzkr26grx3yixjhoagy2dvshcdq

Checking the Submitted window post : https://filfox.info/en/message/bafy2bzacealn3r7midsvnquxebemxunmxumzkr26grx3yixjhoagy2dvshcdq

Seems fine to me : Method : SubmitWindowedPoSt

But still all my sectors have not recovered,
does anybody have an idea what happened?
trying to find an error, but seems fine to me ...

Daemon: 1.4.2+mainnet+git.61eb91ff3.dirty+api1.0.1
Local: lotus-miner version 1.4.2+mainnet+git.61eb91ff3.dirty

thank you,
Angelo

[Angelo-gh3990]

Posted on slack too, if I find the solution I'll post it here too.

[Angelo-gh3990]

jennijuju 1 hour ago
Run lotus-miner pprof goroutines | grep 'FilGenerateWindowPost.func1 ' | wc -l for me please?

Angelo 1 hour ago
lotus-miner pprof goroutines | grep 'FilGenerateWindowPost.func1 ' | wc -l
output = 0

Angelo 1 hour ago
@jennijuju, thanks for your reply, what are you trying to find out?

jennijuju 39 minutes ago
Ellegalargument, do a lotus state replay over that message?

Angelo 35 minutes ago
root@daemon:~# lotus state replay bafy2bzacebqpdaiezajene5djqdhriuypsxtkkon62q6pam2sz6pw5wesgyry
Replay receipt:
Exit code: 16
Return:
Gas Used: 758653
Total Message Cost: 999704704709519176
Error message: "invalid deadline 0 at epoch 512049, expected 1 (RetCode=16)"

Angelo 34 minutes ago
same for the other one:
root@daemon:~# lotus state replay bafy2bzacealn3r7midsvnquxebemxunmxumzkr26grx3yixjhoagy2dvshcdq
Replay receipt:
Exit code: 16
Return:
Gas Used: 757353
Total Message Cost: 928234382559498522
Error message: "invalid deadline 0 at epoch 514976, expected 2 (RetCode=16)"

jennijuju 33 minutes ago
oh yah- it’s submitted to the chain to late

jennijuju 32 minutes ago
Have you been monitoring your mempool?

Angelo 1 minute ago
I see, I'll check it tomorrow during the WinPost ... The mempool is empty when I checked, but I checked after ...

Angelo < 1 minute ago
@jennijuju thanks for helping me identify the problem 🙂