Filecoin Deal Flow, or "How do I store data from my smart contract?"

[hammertoe]

One of the very common questions we are getting asked during the Spacewarp Hackathon is "How do I store data with Filecoin?" or "How to I publish a storage deal?". This might seem like a simple question, as Filecoin is a storage network, and FEVM allows you to write smart contracts – however the answer is not quite straightforward. In this post I hope to give a bit of background to explain this, and answer the question:

TL;DR: Currently you cannot directly store data from a smart contract, instead you have to incentivise humans to do this, and the smart contract can then verify it has happened and then reward the good human

Background

The Filecoin network is designed to store large volumes of data. Currently the network has a committed storage capacity of approx 15EB, which is the equivalent of 1% of the total data storage capacity worldwide. In order to facilitate people wanting to store large volumes of data, the actual process by which data is stored is very flexible – if you have petabytes to store, then you might be shipping hard drives directly to Storage Providers.

Much of the process of storing data on Filecoin happens off-chain

If you have used the Lotus CLI to create storage deals and store data on the network, or used storage aggregators like Estuary or web3.storage, then you might not realise that actually much of the process of storing data on the Filecoin network actually happens off-chain. It is actually not until a storage provider accepts a storage deal and publishes it that the blockchain knows about the storage deal.

Below is a diagram that shows the flow of a client wanting to store data and what parts happen on-chain and off-chain.

Credit: @truckerfling @hammertoe

One very important point to keep in mind with Filecoin is that you can never store or access the stored data from directly within a smart contract. This might at first seem like an odd thing for a smart contract system on a storage network, but there is a very good reason for this. Smart contracts require fees to be paid (“gas”) for each transaction. Those fees are proportional to the execution time and the storage requirements of the smart contract. Filecoin is designed to be able to handle storing gigabytes of data (most storage providers have a “sector size” of 32GB) and so are looking for storage deals in that order of magnitude. With this in mind, the gas cost of processing that data would be exorbitant.

With this in mind, the FVM can be used to automate, incentivise and control the process by which the storage deals for this data are made.

There are a few different ways to architect a smart contract that wants to store data on the Filecoin network...

Current method to store data with FEVM: Incentivising Storage Using Bounties

Currently, the easiest way to get data stored on the Filecoin network is to use "deal bounties". An example of this is provided in the DealRewarder.sol contract example, this is also provided in the Hardhat Starter Kit.

In this model all deal making happens outside of the smart contact. The smart contract itself does not initiate the making of any deals itself, but incentivizes other parties to make those deals and supply the resultant deal ID back to the smart contract to verify.

A simple deal bounty contract consists of a list of the data CIDs that it incentivizes to store on Filecoin. Once a storage deal is made for the listed data, the data bounty hunter can claim the data bounty by providing the deal ID. The contract will check with the Filecoin storage market to confirm whether the supplied deal ID is activated and stores the claimed data. Once validated, the deal bounty contract will pay the bounty hunter out. Below is a diagram to illustrate this:

Credit: @hammertoe @ZenGround0 https://github.com/lotus-web3/deal-bounty-contract

A demo of this approach and a walk through of the example Deal Rewarder contract was shown in the Space Warp summit:

While the Deal Rewarder contract can be used in many use cases, we see strong interest in it being used for data DAOs. Here is an additional resource of a Data DAO Template using Deal Bounty contracts, by community contributor @RishikeshKale: https://github.com/rk-rishikesh/DataDAO/tree/main/contracts/data-dao

Future method to store data with FEVM: Client contract makes storage deals with Storage Provider directly

In the coming week, a new method of storing data directly from a smart contract will be enabled by allowing the Storage Provider (SP) to complete a storage deal directly with a smart contract. The key constraint today is that once an SP publishes a deal proposal on-chain via calling PublishStorageDeal, it requires a client account to authenticate the message. A SP cannot authenticate the message with a client contract today.

The proposed Client Contract method involves the built-in Market Actor delegating authorization of a deal to an arbitrary smart contract. This allows:

1. The user to input their deal proposal into a smart contract
2. Send deal proposal to the SP
3. SP accepts and calls PublishStorageDeal to the Market Actor
4. Deal is published on-chain and SP receives a DealID
5. Simultaneously, Market Actor authenticates the publication of the storage deal with the Client Contract.

An example of this approach is currently being worked on by @ZenGround0 at https://github.com/lotus-web3/client-contract

Future method to store data with FEVM: Participating Storage Provider (PSP) contract makes storage deals with data DAO directly

To publish a storage deal on-chain, a SP must call PublishStorageDeal. Typically only a SP who is not a contract, can do so from their control address, to ensure that they have committed to the deal before having to undertake the effort to store the data. The Participating Storage Provider method proposes that the SP replaces their control address with a smart contract. This is estimated to be available by mid February, after the hackathon (nothing stopping devs from building this during the hackathon as the WIP piece below is outlined in a GitHub issue!) .

Here is the process (referenced heavily from this issue):

1. A DataDAO allows miners to register to participate as storage providers in the DAO.
2. When an SP registers, the DataDAO creates a “Participating Storage Provider” (PSP) contract via CREATE2 and returns its address. This is the gateway for the miner to participate in the DAO.
3. The SP adds the PSP as a control address via [ChangeWorkerAddress](https://github.com/longfeiWan9/BuiltIn-Actor-API/blob/main/Miner.md#changeworkeraddress) to its Miner actor, thus allowing it to call [PublishStorageDeals](https://github.com/longfeiWan9/BuiltIn-Actor-API/blob/main/Storage%20Market%20Actor.md#publishstoragedeals) on its behalf.
4. When the DataDAO decides it wants to make a deal with the miner, it internally creates a deal proposal, which emits an event to the SP (via a deal proposer event FRC which is WIP). SP and Client then decide on the deal proposal acceptance.
5. When N number of deals have been made, the SP decides to commit to storing them, and sends a message to the PSP, which calls PSD on its behalf with the accumulated deal proposals.
6. The SP downloads the data through whatever mechanism the DataDAO designates (IPFS, Boost, HTTP, etc.)
7. The SP completes the sealing process and the deal is active.

[PatrickAlphaC]

Great writeup, thanks for posting.

[hammertoe]

Hey folks, just to say I've updated the post above with some more information about future approaches to storage deal creation, and a diagram showing the current way using bounties.

[trruckerfling]

@hammertoe image error above :/