A new ChainIndexer that subsumes that existing MsgIndex, EventIndex and TransactionIndex

[aarshkshah1992]

Summary

This issue is for the implementation of a new ChainIndexer in Lotus that will replace and subsume the existing MsgIndex, EventsIndex, and EthTxHashIndex, which are currently fragmented across multiple databases and have several known issues documented in filecoin-project/lotus#12293.

Tasks

Beta Give feedback

1. Land ChainIndexer implementation at feat: a new ChainIndexer to index tipsets, messages and events #12421
2. Land feat: recompute tipset to generate missing events if event indexing is enabled #12463
3. Land Migration/Backfilling/Diagnostics tooling for the ChainIndexer at feat: migration("re-indexing"), backfilling and diasgnostics tooling for the ChainIndexer #12450
4. Land feat: ChainIndexer: improve sqllite query handling #12485
5. sqlite: adjust page size #12489
   +0
   
6. Work with Glif to deploy the ChainIndexer on their FEVM archival node and re-index the entire archived chain history in the ChainIndexer. OKU should then be able to use that node for their app. A green light from Glif & OKU is what we need here to then ship this out to all users.
7. Write a document explaining the mechanics of re-indexing/backfilling/diagnostics/inspection of the ChainIndexer and explain how RPC providers can use the RPCs/CLI commands being built out in feat: migration("re-indexing"), backfilling and diasgnostics tooling for the ChainIndexer #12450 for all of those use cases.
8. Document the Config changes introduced in feat: a new ChainIndexer to index tipsets, messages and events #12421 so users understand the knobs they can turn on/off for the ChainIndexer.
9. Document clean up process for the existing MsgIndex,EthTxIndex and EventIndex (should be as simple as getting rid of the Sqlite DBs in the Lotus repo).
10. Determine release timing/process in light of nv24

Key Features

The ChainIndexer offers the following key features:

• Indexes all necessary state in a single database.
• We now index a "tipset" and the relevant state changes caused by a tipset which makes the Indexed state easy to reason about and more aligned with how we persist state in the Chainstore.
• Implements snapshot hydration.
• Implements automated config driven Index garbage collection (GC).
• Provides automated backfilling.
• Offers simplified configuration.
• Wraps the asynchronous indexing in a synchronous READ API for RPC endpoints to consume to avoid off by N errors caused by async Indexing.

Here are the next implementation steps going ahead to get this PR in a ready for review/ready for testing state.

Switch RPC APIs to use the Chain Index

• The Filecoin and ETH RPC APIs will switch to using the ChainIndexer instead of the MsgIndex, EthTxHashIndex and EventsIndex.
• The EventFilterManager will read events from the ChainIndexer and prefill all registered filters rather than depending on the Indexer to do the pre-filling of filters.
• The ChainIndexer will listen to Mpool message addition updates to index the corresponding ETH Tx Hash. The EthTxHashManager will no longer be used for this.

Read APIs Should Account for the Async Nature of Indexing

• All APIs that read from the index will wait for the current head in the chainstore to be indexed if their first read attempt fails and then retry the read before returning a response to the user.
  • Note that we're only waiting for an already produced tipset in our chainstore to be indexed here; we're not waiting for a new tipset to be produced. This should work well in practise but if it times out, it points to another underlying problem in our chainstore <-> indexing pathway that should be investigated.
• This is a workaround to handle asynchronous indexing in Filecoin.
• For events, it should be noted that indexing the current head T only indexes events in T-1 because of deferred execution.

ETH RPC APIs Should Only Expose Executed Tipsets and Messages

• As part of this work, we should review all ETH RPC APIs to ensure they only expose and respond to requests for executed tipsets and transactions. This is because Filecoin uses a deferred execution model, unlike Ethereum:
• In Filecoin, messages included in tipset T are executed in tipset T + 1.
• In Ethereum, messages included in tipset T are also executed in tipset T.
• Exposing messages and tipsets that have not been executed yet via ETH RPC APIs in Lotus causes errors when users ask for the corresponding execution state, receipts, or events for those tipsets/messages because they do not yet exist.
• There have been proposals for an even more conservative implementation where ETH RPC APIs only expose "finalized" tipsets and messages post-F3. However, it remains to be determined how well this would work in practice, given that clients might end up waiting for up to 3 epochs (90 seconds) for already included messages in the worst case.
• For now, we should go ahead with exposing all(even non-finalised) executed tipsets/messages and revisit the implementation after F3 ships.

Removing Re-orged Tipsets That Are No Longer Part of the Canonical Chain

• The ChainIndexer will periodically prune all permanently re-orged/reverted tipsets from the index. It can do this by simply pruning all tipsets at a height less than (current head - finality policy - some buffer).
• The use of foreign key-based cascading deletes in the DDL will greatly simplify this implementation. By simply deleting a tipset from the index, all associated indexed state will be deleted from the DB. See SQLite Foreign Keys for more information.

Garbage Collection

• Garbage collection (GC) will be configuration-driven. Users can specify how much history they want to retain, and the ChainIndexer can perform periodic GC based on this configuration.
• GC should be straightforward in the ChainIndexer because of the use of FOREIGN KEY ON CASCADE DELETES, as described in SQLite Foreign Keys.
• When a tipset is deleted, all associated indexed state will be automatically deleted from the DB due to the cascading delete behavior.

Snapshot Hydration

• When a node is synced from a snapshot, the index should be completely deleted, and a new index should be hydrated from the snapshot.
• It's important to note that snapshots don't contain events. In order to hydrate events in the index, messages in the tipset will have to be re-executed.

Automated Backfilling

• When a Lotus node starts up, it performs the following steps:
  • Looks up the latest non-reverted tipset in the ChainIndex for which the corresponding state exists in the statestore.
  • Instantiates the Observer with that tipset as the current head.
  • Starts the Observer.
• This process ensures that the ChainIndexer will observe the (Apply, Revert) path between its last non-reverted indexed tipset and the current heaviest tipset in the chainstore before processing real-time updates, effectively performing automated backfilling.
• One challenge arises for a niche use case when an RPC provider toggles the Indexing flag to ON after keeping it OFF for an extended period or for the first time. In such cases, the backfilling backlog could interfere with indexing real-time tipset changes, potentially impacting RPC queries that primarily target state at or near the head.
• To address this issue, a configuration option can be exposed that allows such users to disable automated backfilling if their primary focus is serving RPC queries for new tipsets after enabling indexing.

Simplify Indexing Config

• The current indexing configuration in Lotus is extremely complex, with partial indexing options that make it difficult for node operators to understand the state they are indexing or should be indexing.
• To improve the user experience and simplify the implementation, the current config will be replaced with a simple "Indexing ON/Indexing OFF" switch. Users either index everything that Lotus needs to provide fast RPC responses or index nothing.
• The niche use case of "Index X but do not Index Y" will no longer be supported.

Migration from Old Indices to the New ChainIndex

• Develop a lotus-shed utility that allows users to migrate existing indices to the new ChainIndexer database. This command should only be executed when the Lotus node is offline to ensure data consistency and avoid potential conflicts.
• When a Lotus node starts up, it should bypass any migration or backfilling processes and directly begin indexing new tipsets in the ChainIndexer. This approach offers several benefits:

1. Users can migrate the historical index at their own pace without incurring a performance penalty during node startup.
2. The node can quickly respond to queries for new tipsets since indexing for these tipsets commences as soon as the node is operational.

• By decoupling the migration process from the node startup, users gain flexibility in managing the transition to the new indexing system while maintaining optimal performance for real-time tipset indexing.