Spark v1.5: limited support for DDO deals

[bajtos]

There is an increasing urgency for Spark to support DDO deals. The Spark v2 architecture requires changes in miner SW; it will take months until the network meets Spark v2 requirements.

Spark v1.5 is a short-term stop-gap solution which retrofits DDO deals onto the existing Spark v1 database of eligible deals in a backwards-compatible way that preserves current RSR for f05 deals.

Consult the following documents for more details:

• Spark & DDO deals
• Spark DDO hack

Tasks:

• Enhance fil-deal-ingester to store PieceCID and PieceSize alongside Label CID in the database of eligible deals. Also, add a new column to the database of eligible deals to distinguish between f05 and DDO deals.
  → Extend eligible deals schema to enable adding DDO deals filecoin-station/fil-deal-ingester#29
• Build deal-observer for DDO as planned for Spark v2. This will produce a list of (minerId, clientId, PieceCID, PieceSize) rows in an SQL table.
  → Deal Observer: Research & Design Spec #171
  → Deal Observer: initial implementation (deal activation events) #172
• Add a PayloadCID column to deal-observer DB and set it to NULL for newly ingested deals.
• Periodically scan the table produced by spark-observer and for each deal that does not have PayloadCID set, ask piece-indexer for a sample payload block CID and store the CID in our table if found. If not, then we need to retry the query, because it can take some time until IPNI advertisements are processed for the new deal. We can stop retrying after some configured time, e.g. 3 days.
• Periodically cross-check the list of f05 deals created by fil-deal-ingester with the list of deals created by deal-observer. Find deals that are in the spark-observer database only and which have PayloadCID value filled in. Add these deals to the “master” table of deals eligible for retrieval checking, and flag these deals as coming from spark-observer.
  • There should be a lot of overlap between the list produced by fil-deal-ingester and deal-observer, because f05 deals will be ingested by both services.