Real-time view of all storage deals (including DDO)

[bajtos]

eta: 2024-09-30

We currently have a manual process to process StateMarketDeals, extract deals that are expected to be retrievable, and update Spark's database. This process does not support recently introduced DirectDataOnboarding deals (see FIP 76).

Let's implement a new solution that will observe actor events (see FIP 83) and update Spark's database of retrievable deals in real-time.

Resources:

• Research notes:
  https://docs.google.com/document/d/1YvFYYUwP6VibKyTeHLd89mrkB1S5sVjq9ydFCGbBfrw
• Spark v2 design & migration:
  https://spacemeridian.notion.site/Spark-v2-design-migration-115cdd5cccdb804ca4d1e0694c613318?pvs=4

We need at least the following data for each deal:

• miner_id
• client_id
• piece_cid
• piece_size

To link deals to allocators, we need to maintain the list of client<->allocator links by listening to on-chain actor events emitted by the AddVerifiedClient method. We should build this component as part of this task.

Subtasks

• Deal Observer: Research & Design Spec #171
• Deal Observer: initial implementation (deal activation events) #172
• Deal Observer: deal expiration/slashing events
• Deal Observer: deal extension events
• Deal Observer: allocator-client events

[bajtos]

Deals can be extended, see e.g. here:

• https://docs.filswan.com/swan-storage-market/core-modules/extend-datacap-terms-service
• https://lotus.filecoin.io/lotus/manage/claim-extension/