feat: migration("re-indexing"), backfilling and diasgnostics tooling for the `ChainIndexer` #12450

aarshkshah1992 · 2024-09-11T15:23:06Z

ChainIndexer Migration and Diagnostics Tooling

This PR implements the "migration" (really re-indexing / backfilling), and diagnostics tooling for the ChainIndexer implemented in PR #12450, and is part of the work for #12453. This tooling takes the form of both RPC APIs on the daemon and lotus-shed CLI commands.

Re-indexing Process

The re-indexing tool enables clients to index their entire existing ChainState in the ChainIndexer. This process is necessary due to the removal of the existing MsgIndex, EthTxIndex, and EventIndex from Lotus.

Why Re-index Instead of Migrate?

We've chosen to re-index rather than migrate data from existing indices for two primary reasons:

Known issues: The existing indices have multiple known problems, and migrating could perpetuate incorrect index entries.
Lack of garbage collection: Existing indices contain many entries for which the corresponding tipset messages/events no longer exist in the ChainStore due to splitstore GC.

Instead, we're re-indexing the Chainstore/Chainstate on the node into the ChainIndexer. This ensures that all re-indexed entries have gone through the indexing logic of the new ChainIndexer and that the Index is in sync/reflects the actual contents of the Chainstore/Chainstate post re-indexing.

Diagnostics Tooling

This PR introduces diagnostic tools for detecting corrupt Index entries at specific epochs or epoch ranges.

While this PR implements functionality for optionally backfilling missing Index entries, it does not yet include the capability to "repair" corrupted Indexed entries. The repair functionality will be introduced in a subsequent PR. This approach allows us to first gather and analyze user reports, helping us understand the types and causes of corrupted Indexed entries(and if all they exist in the new ChainIndexer) before implementing repair mechanisms.

Core API

The fundamental building block for this tooling is the following RPC API:

type IndexValidation struct {
	TipsetKey string
	Height    uint64

	TotalMessages  uint64
	TotalEvents    uint64
	EventsReverted bool

	Backfilled bool
}

func (si *SqliteIndexer) ChainValidateIndex(ctx context.Context, epoch abi.ChainEpoch, backfill bool) (*types.IndexValidation, error)

This API has the following features:

Optionally backfills the Index with a tipset on the canonical chain for the given epoch if it is absent in the Index
Returns some aggregated stats for an indexed entry for diagnostics/inspection
Reports errors/corrupted indexed entries at the given epoch. Forms of Index corruption that can be diagnosed includes:
- Presence of multiple non-reverted tipsets at the given epoch
- Complete absence of a non-reverted tipset at the given epoch that does contain reverted tipsets
- Mismatch between the Chainstore state and the Indexed entries (tipset messages/events)
- Incorrect Indexing of null rounds at the given epoch

`lotus-shed` CLI tooling

The lotus-shed CLI tooling for both re-indexing/backfilling and diagnostics can then invoke this RPC API over epoch ranges. The corresponding lotus-shed backfill index [from, to] and lotus-shed inspect index [from, to] can then backfill/inspect/diagnose the Index for the given epoch ranges.

TODO

lotus-shed commands
RPC API should compare the Indexed entries with the chain store to detect a mismatch between indexed content and chain store content
Fill out all the diagnostics information in the RPC response by queying the messages and events tables in the Index.
automated tests

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

chainindex/api.go

aarshkshah1992 · 2024-09-11T15:32:03Z

chainindex/api.go

+	return &types.IndexValidation{
+		TipsetKey:  ts.Key().String(),
+		Height:     uint64(ts.Height()),
+		Backfilled: true,


Fill out the other fields here using SQL queries.

chain/types/tipset.go

chainindex/indexer.go

chainindex/api.go

chainindex/ddls.go

aarshkshah1992 · 2024-09-12T06:55:37Z

@rvagg Would be great to have a first round of review here when you get the time.

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

chain/index/api.go

rvagg · 2024-09-17T06:54:25Z

chain/index/api.go

+	return revertedCount, nonRevertedCount, nil
+}
+
+func (si *SqliteIndexer) ChainValidateIndex(ctx context.Context, epoch abi.ChainEpoch, backfill bool) (*types.IndexValidation, error) {


if we did this all in a single transaction, would that remove the need for a write lock?

IIUC, transaction provides atomicity, not exclusivity. Imagine this scenario:

Backfill is called for epoch e -> it reads tipset ts at that epoch

Revertexecutes for ts and marks the tipset as reverted in TX1

Backfill executes for ts and marks it as applied in TX2

TX1 is committed first and then TX2 is comitted.

Basically, it's hard to reason about state here without this lock.

@rvagg

One work-around here it to only allow backfilling for "safe" epochs i.e. <=head - 30.

Then, the rewrite can only ever happen in an edge case.

chain/index/api.go

chain/index/ddls.go

rvagg · 2024-09-17T07:04:05Z

chain/index/api.go

+
+	// fetch the non-reverted tipset at this epoch
+	var indexedTsKeyCidBytes []byte
+	err = si.stmts.getNonRevertedTipsetAtHeightStmt.QueryRowContext(ctx, epoch).Scan(&indexedTsKeyCidBytes)


see the LIMIT 1 below if you're only expecting one result

rvagg · 2024-09-17T07:05:51Z

chain/index/api.go

+	}
+
+	return &types.IndexValidation{
+		TipsetKey: expectedTs.Key().String(),


see below, I think just the key would be good here, also note capitalisation of TipSetKey

rvagg · 2024-09-17T07:06:45Z

chain/index/api.go

+	return &types.IndexValidation{
+		TipsetKey:  ts.Key().String(),
+		Height:     uint64(ts.Height()),
+		Backfilled: true,


todo: other fields

* feat: finish todos of validation api * feat: add indexed data verification with chain store * feat: address comments and finish TODO * fix: build issue * address comments * fix: ci issue

aarshkshah1992 · 2024-09-17T07:48:50Z

Thanks for all your work here @akaladarshi 👍

Co-authored-by: Rod Vagg <[email protected]>

akaladarshi · 2024-09-17T11:31:53Z

chain/types/index.go

-	NonRevertedEventsCount  uint64
-	Backfilled              bool
+	IndexedMessagesCount uint64
+	IndexedEventsCount   uint64


Should we rename this to IndexedNonRevertedMsgCount and IndexedNonRevertedEventsCount to be precise?

@akaladarshi That is an implementation detail that will confuse users. They're only asking for the indexed entries at a specific epoch here and so ofcourse expect only non-reverted tipsets back.

* feat: add lotus-shed command for backfilling chain indexer * feat: add lotus-shed command for inspecting the chain indexer * feat: use single lotus-shed command to inspect and backfill * fix: remove the unused queries * small changes * add change log

aarshkshah1992 · 2024-09-19T09:10:29Z

@BigLep This is now ready for review. Working on a user guide for this tooling as discussed offline.

aarshkshah1992 · 2024-09-19T09:10:56Z

@rvagg This is now in a good shape and has been tested on a calibnet node. Please take a look when you get the time.

…tion-api

akaladarshi · 2024-09-19T11:41:56Z

cmd/lotus-shed/chain_index.go

+				_, _ = fmt.Fprintf(cctx.App.Writer, "chain index validation progress: %.2f%%\n", progress)
+			}
+
+			indexValidateResp, err := api.ChainValidateIndex(ctx, abi.ChainEpoch(epoch), backfill)


@aarshkshah1992 what will happen if user provided chainHead as the fromEpoch?

getNextTipset will error out, I think.

akaladarshi · 2024-09-19T11:42:03Z

cmd/lotus-shed/chain_index.go

+				backfilledEpochs = append(backfilledEpochs, epoch)
+			}
+
+			if output {


I think we should also log the result of each epoch.

Never mind, didn't see the new changes.

akaladarshi · 2024-09-19T11:55:32Z

cmd/lotus-shed/chain_index.go

+				_, _ = fmt.Fprintf(cctx.App.Writer, "✓ Epoch %d (%s)\n", epoch, string(jsonData))
+			}
+		}
+


Add a log for completion as well.

I think we can also print how many epochs were success out of total epoch or something like that.

akaladarshi · 2024-09-19T11:56:42Z

cmd/lotus-shed/chain_index.go

+				if strings.Contains(err.Error(), "retry") {
+					log.Warnf("epoch %d; failed to validate index with re-tryable error: %s. retrying...", epoch, err)
+					epoch++ // Increment epoch to retry the same epoch
+					continue


Should we sleep here for sometime because retry will happen due to re-orgs and that might take sometime?.

akaladarshi · 2024-09-19T12:17:54Z

cmd/lotus-shed/chain_index.go

+			// is it a null round ?
+			if indexValidateResp.IsNullRound {
+				if logGood {
+					_, _ = fmt.Fprintf(cctx.App.Writer, "✓ Epoch %d; null round\n", epoch)
+				}
+				continue
+			}
+
+			// log success
+			if logGood {
+				jsonData, err := json.Marshal(indexValidateResp)
+				if err != nil {
+					return fmt.Errorf("failed to marshal results to JSON: %w", err)
+				}
+
+				_, _ = fmt.Fprintf(cctx.App.Writer, "✓ Epoch %d (%s)\n", epoch, string(jsonData))
+			}


Suggested change

// is it a null round ?

if indexValidateResp.IsNullRound {

if logGood {

_, _ = fmt.Fprintf(cctx.App.Writer, "✓ Epoch %d; null round\n", epoch)

}

continue

}

// log success

if logGood {

jsonData, err := json.Marshal(indexValidateResp)

if err != nil {

return fmt.Errorf("failed to marshal results to JSON: %w", err)

}

_, _ = fmt.Fprintf(cctx.App.Writer, "✓ Epoch %d (%s)\n", epoch, string(jsonData))

}

// log success

if logGood {

if indexValidateResp.IsNullRound {

_, _ = fmt.Fprintf(cctx.App.Writer, "✓ Epoch %d; null round\n", epoch)

} else {

jsonData, err := json.Marshal(indexValidateResp)

if err != nil {

return fmt.Errorf("failed to marshal results to JSON: %w", err)

}

_, _ = fmt.Fprintf(cctx.App.Writer, "✓ Epoch %d (%s)\n", epoch, string(jsonData))

}

}

github-actions bot previously requested changes Sep 11, 2024

View reviewed changes