Revert blocks during reorg

[junderw]

Re-index not required (but highly recommended, to remove incorrect old data from the DB)

---

Take 2

Alternative to #111

[mononaut]

Concept ACK

This definitely feels like the right approach, and stepping through the code the implementation all makes sense to me.

I'll upgrade to a full ACK once I figure out how to test it properly :)

[junderw]

Never mind. I take back my take back! lol

The channel sendbacks are just for b'I' rows which are included in the cache for assets. Only sending back AssetIds from b'I' rows is correct, Since we want to remove them from the cache.

[mononaut]

Tested NACK @ [737d725]

I've been running this on mainnet locally for a few days, and not observed any problems at all.

To test the reorg behavior more thoroughly, I scripted the following reproducible scenarios with a pair of regtest nodes:

#	Scenario	Master branch	comments	This PR	comments
1	Simple short reorg	⁉️	reorged coinbase remains in txstore, but is filtered out of address results	✅	reorged coinbases removed from both txstore and history dbs
2	Transaction reorged and reconfirmed in another block at the same height and index	✅		✅
3	Transaction reorged and reconfirmed in another block at the same height but a different index	❌	transaction is filtered out of /address/.../txs, but double-counted towards txos in chain_stats	✅
4	Transaction reorged and reconfirmed in a later block at the same index	✅		✅
5	Long-range reorg (10 blocks)	✅		❌🔥	electrs crashes

For short reorgs, this PR fixes the immediate issue of transactions confirmed at the same height and different indexes being double-counted.

However, long reorgs reliably crash electrs on this PR, sometimes into an irrecoverable state:

DEBUG - applying 11 new headers from height 103
DEBUG - reorg of 10 blocks detected
DEBUG - hash=73f96de01d42ae6349ae1c2e468b9ce6ae19cd0417f8864fc4ee87a799cbecc8 height=103 @ 2024-12-24T22:06:42Z (10 left to index)
TRACE - [THREAD] GETHASHMAP INSERT | bitcoind_fetcher-34 ThreadId(291)
TRACE - [THREAD] START WORK        | bitcoind_fetcher-34 ThreadId(291)
TRACE - [THREAD] FINISHED WORK     | bitcoind_fetcher-34 ThreadId(291)
TRACE - [THREAD] GETHASHMAP REMOVE | bitcoind_fetcher-34 ThreadId(291)
TRACE - [THREAD] HASHMAP REMOVED   | bitcoind_fetcher-34 ThreadId(291)
DEBUG - Deleting 10 blocks to Indexer
DEBUG - deleting 110 rows from RocksDB { path: "/electrs/db/regtest/newindex/txstore" }
DEBUG - hash=73f96de01d42ae6349ae1c2e468b9ce6ae19cd0417f8864fc4ee87a799cbecc8 height=103 @ 2024-12-24T22:06:42Z (10 left to index)
TRACE - [THREAD] GETHASHMAP INSERT | bitcoind_fetcher-35 ThreadId(292)
TRACE - [THREAD] START WORK        | bitcoind_fetcher-35 ThreadId(292)
TRACE - [THREAD] FINISHED WORK     | bitcoind_fetcher-35 ThreadId(292)
TRACE - [THREAD] GETHASHMAP REMOVE | bitcoind_fetcher-35 ThreadId(292)
DEBUG - Indexing (Deleting) 10 blocks with Indexer
TRACE - [THREAD] HASHMAP REMOVED   | bitcoind_fetcher-35 ThreadId(292)
thread 'thread 'lookup-txo-12lookup-txo-8' panicked at ' panicked at src/new_index/schema.rssrc/new_index/schema.rs:1448:29:
missing txo 0c674ca8df296ab312e4125be3032d9904221f0b646a89d9bab67092996b8b09:0 in DB { db: RocksDB { path: "/electrs/db/regtest/newindex/txstore" } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'thread 'thread 'lookup-txo-2lookup-txo-15thread ':1448:' panicked at src/new_index/schema.rs:1448:29thread 'lookup-txo-7' panicked at lookup-txo-3src/new_index/schema.rs:' panicked at 1448' panicked at 29:src/new_index/schema.rs:
missing txo 5d94c442d19918c3002bbcc641a09a84370fcb36807f69fd7b81c4d0ccd712fa:1 in DB { db: RocksDB { path: "/electrs/db/regtest/newindex/txstore" } }29lookup-txo-4thread 'thread '

(note: I added a couple of extra debug logs to this trace, so line numbers etc might not line up exactly)

the panic seems to be triggered by the reorg handling itself, rather than any collision or race condition with a simultaneous API request. My guess is that we're trying to lookup_txos after the corresponding transactions have already been deleted from the txstore, but I haven't quite teased out the exact sequence of events.

[junderw]

This should remove all issues. We will sequentially fetch 100 blocks at a time from bitcoind in reverse header order and delete them in order from the tip to the stem... so it shouldn't cause issues...

(crosses fingers)

[mononaut]

Tested ACK @ [a454994]

Still no problems running locally on mainnet, and now it passes all of my scripted regtest reorg scenarios:

#	Scenario	Master branch	comments	This PR	comments
1	Simple short reorg	⁉️	reorged coinbase remains in txstore, but is filtered out of address results	✅	reorged coinbases removed from both txstore and history dbs
2	Transaction reorged and reconfirmed in another block at the same height and index	✅		✅
3	Transaction reorged and reconfirmed in another block at the same height but a different index	❌	transaction is filtered out of /address/.../txs, but double-counted towards txos in chain_stats	✅
4	Transaction reorged and reconfirmed in a later block at the same index	✅		✅
5	Long-range reorg (10 blocks) with mempool re-entry	✅		✅

[junderw]

2 minds are better than one, good catch on the flipped order.

LGTM on the changes you made.