node: wrong state root after 100 blocks unwind

[canepat]

While using the staged_pipeline utility to check the unwind procedure on mainnet, this sequence of commands:

staged_pipeline --exclusive --datadir ~/silkworm reset_to_download --keep_senders --force
staged_pipeline --exclusive --datadir ~/silkworm debug_unwind --height 1000000 --stop_before_stage HistoryIndex --step 100 --force

produces a state root mismatch at the end of InterHashes stage:

  INFO [12-13|23:35:57.238 UTC] Debug unwind datadir: ~/silkworm
  INFO [12-13|23:36:03.298 UTC] Reopen snapshot repository folder: ~/silkworm/snapshots
  INFO [12-13|23:36:05.043 UTC] Total reopened bundles: 72 max block available: 20'474'999
  INFO [12-13|23:36:05.043 UTC] Debug unwind: forward to=1'000'000 START
  INFO [12-13|23:36:05.043 UTC] ExecutionPipeline                        Forward start
  INFO [12-13|23:36:05.043 UTC] [5/13 Execution]                         op=kForward from=0 to=1000000 span=1000000 
  INFO [12-13|23:36:35.045 UTC] [5/13 Execution]                         block=808791 blocks/s=26959 txns/s=37191 Mgas/s=1525 
  INFO [12-13|23:36:48.233 UTC] Flushed state                            size=41.53 MB in=597.315ms 
  INFO [12-13|23:36:48.489 UTC] [5/13 Execution] commit                  batch time=254.726ms 
  INFO [12-13|23:36:48.490 UTC] [5/13 Execution]                         op=Forward done=43.446s 
  INFO [12-13|23:36:48.490 UTC] [6/13 HashState]                         op=kForward from=0 to=1000000 span=1000000 
  INFO [12-13|23:36:48.490 UTC] [6/13 HashState]                         clearing=HashedAccount 
  INFO [12-13|23:36:48.490 UTC] [6/13 HashState]                         clearing=HashedStorage 
  INFO [12-13|23:36:48.490 UTC] [6/13 HashState]                         clearing=HashedCodeHash 
  INFO [12-13|23:36:49.177 UTC] [6/13 HashState]                         op=Forward done=686.119ms 
  INFO [12-13|23:36:49.177 UTC] [7/13 IntermediateHashes] begin          op=kForward from=0 to=1000000 span=1000000 
  INFO [12-13|23:36:49.177 UTC] [7/13 IntermediateHashes]                clearing=TrieAccount 
  INFO [12-13|23:36:49.177 UTC] [7/13 IntermediateHashes]                clearing=TrieStorage 
  INFO [12-13|23:36:49.696 UTC] [7/13 IntermediateHashes]                op=Forward done=519.858ms 
  WARN [12-13|23:36:49.696 UTC] Stopping ...                             STOP_BEFORE_STAGE=HistoryIndex hit=true 
  INFO [12-13|23:36:49.697 UTC] ExecutionPipeline                        Forward done
  INFO [12-13|23:36:49.697 UTC] Debug unwind: forward to=1'000'000 END
  INFO [12-13|23:36:49.697 UTC] Debug unwind: unwind down to block=999'900 START
  INFO [12-13|23:36:49.697 UTC] ExecutionPipeline                        Unwind start
  INFO [12-13|23:36:49.697 UTC] [7/13 HashState]                         op=kUnwind from=1000000 to=999900 span=100 
  INFO [12-13|23:36:49.707 UTC] [8/13 IntermediateHashes] begin          op=kUnwind from=1000000 to=999900 span=100 
 ERROR [12-13|23:36:49.720 UTC] Wrong trie root                          expected=0x33d8f415fd268da5169ac45e5f8e07fe710c23e4f4e2a318a9f781bd196f2066 got=0xd418c5ebada98e1a1bad2cec92153ca2baaa6cb6a543b23ae4968307b7192bc9 
 ERROR [12-13|23:36:49.720 UTC] [8/13 IntermediateHashes]                function=unwind exception=kWrongStateRoot 
 ERROR [12-13|23:36:49.720 UTC] [8/13 IntermediateHashes]                op=Unwind returned=kWrongStateRoot 
 ERROR [12-13|23:36:49.720 UTC] ExecutionPipeline                        Unwind interrupted due to stage IntermediateHashes failure
Error: unwind failed: kWrongStateRoot

[battlmonstr]

Keeping the bug open, as I understand the bug is not fully fixed, but only workarounded by using a different (slower?) code path.
See TODOs in stage_interhashes.cpp InterHashes::forward and InterHashes:: unwind

[canepat]

Keeping the bug open, as I understand the bug is not fully fixed, but only workarounded by using a different (slower?) code path. See TODOs in stage_interhashes.cpp InterHashes::forward and InterHashes:: unwind

Yep, see also #2596