fix(content): update filecoin piece content #1016
Conversation
|
Hey, while we're doing this, if we end up with the longer description of how to get to CommP, how about we take the opportunity to insert a definition of "Fr32" into the specs because it's going to keep getting asked. @ribasushi was the latest one to ask just 2 days ago. Combining and rephrasing parts of @porcuquine's responses we could come up with something like:
|
yiannisbot
force-pushed
the
pieces-fix
branch
from
94aa3a1
to
13e33f3
Compare
|
@porcuquine @rvagg @ribasushi can you have another look at this after a revision I did to the description? I think I've addressed all the issues discussed above. |
|
|
||
| 2. In order to make a _Filecoin Piece_, the IPLD DAG is serialised into a ["Content-Addressable aRchive" (.car)](https://github.com/ipld/specs/blob/master/block-layer/content-addressable-archives.md#summary) file, which is in raw bytes format. A CAR file is an opaque blob of data that packs together and transfers IPLD nodes. The _Payload CID_ is common between the CAR'ed and un-CAR'ed constructions. This helps later during data retrieval, when data is transferred between the storage client and the storage provider as we discuss later. | ||
|
|
||
| 3. The resulting .car file is _padded_ with extra zero bits in order for the file to make a binary Merkle tree. To achieve a clean binary Merkle Tree the the .car file size has to be in some power of two (^2) size. Therefore, the padding process takes the input file and finds the size above the input size that makes for a power of two size. The gap between the input size and the next power of two size is padded with zeros. Finally, an extra padding process is applied, called `fr32 padding` which adds two (2) zero bits to every 254 bits (to make 256 bits in total). |
There was a problem hiding this comment.
It looks like this para has gone through some iterations and lost the initial ordering in an attempt to make it easier to explain - which I understand, but it's now not quite accurate. The final size needs to be a ^2 but the fr32 padding gets in there before it goes to ^2. So to calculate how much zero padding to add, you have to account for the 254/256 inflation that's going to occur.
I think in an earlier iteration it was suggested to switch the order of the steps and say that fr32 happens first and then inflation to ^2 after that. I think this is the ideal framing because it makes it easier to understand (even though in code it happens the other way around, it doesn't actually matter so it's worth doing what's best for descriptive purposes). So that would mean pulling the last sentence to be first, "A process called fr32 padding which adds ... is applied. Extra padding is applied to the end of the data, adding zero bytes until the resulting blob is a power of two sie.". Something like that.
There was a problem hiding this comment.
Forgot to mention that I've removed the definition of fr32 from here and included it in the Glossary (outstanding PR atm).
There was a problem hiding this comment.
I'm just noticing this now. I'm commenting here without an entirely clear point, but so it doesn't get lost in the context shift:
Fr32 is well-defined in the rust-fil-proofs implementation. As far as I can tell from the glossary, it's not given a substantive definition there. It's use here as an adjective was never particularly well-defined (from what I saw), although some true statements were made justifying the adjective.
Without making a specific suggestion for how to untangle this, let me offer the definition underlying the type called Fr32 in rust-fil-proofs. Fr32 is a 32-bit representation of a field element (which, in our case, is now, is the arithmetic field of BLS12-381). To be well-formed, a value of type Fr32 must actually fit within that field, but this is not enforced by the type system. It is an invariant which must be perserved by correct usage.
In the case of so-called Fr32 padding, two zero bits are inserted 'after' a number requiring at most 254 bits to represent. This guarantees that the result will be Fr32, regardless of the value of the initial 254 bits. This is a 'conservative' technique, since for some initial values, only one bit of zero-padding would actually be required.
It's possible that none of this needs to be spelled out explicitly here. It's not clear that an 'official' definition of the term makes much sense without at least some of this context, though. I've put this comment here rather than there because it's in the context of this conversation that the added detail is obviously relevant.
There was a problem hiding this comment.
I should note that Fr32 is not obviously a concept which belongs in the spec, even though it seems to have made its way into the preferred name for what I historically called 'bit padding'. I don't mind if it does stay in the spec as a glossary term. I mention this just to clarify that its primary definition is as an implementation type with relatively elaborate characteristics.
In particular, it's important to understand that Fr32-padded data represents a smaller set of values than inhabit the Fr32 type.
There was a problem hiding this comment.
@porcuquine this is useful input. I have added your definition above as an extra "important note" in this section (see commit 972763d) and will also add this as a definition in the glossary too, if you don't mind.
I should note that Fr32 is not obviously a concept which belongs in the spec
I generally tend to agree. However, if we don't mention anything specific to Fr32, then we should just mention what is in one of the notes: "any implementation has to make sure that the initial IPLD DAG is serialised and padded so that it gives a clean binary tree, and therefore, calculating the Merkle root out of the resulting blob of data gives the same Piece CID", and leave it to that, without any mention of padding, .car etc.
I suggest leaving the text as is for now, given we mention that all this is Lotus-specific and the alternative would be slightly confusing to describe. Let me know if you're ok with this.
| - Steps 2 and 3 above are specific to the Lotus implementation. The same outcome can be achieved in different ways. However, any implementation has to make sure that the initial IPLD DAG is serialised and padded so that calculating the Merkle root out of the resulting blob of data gives the same **Piece CID**. As long as this is the case, implementations can deviate from the first three steps above. | ||
| - `Fr32` is a 32-bit representation of a field element (which, in our case, is the arithmetic field of BLS12-381). To be well-formed, a value of type `Fr32` must _actually_ fit within that field, but this is not enforced by the type system. It is an invariant which must be perserved by correct usage. | ||
| In the case of so-called `Fr32 padding`, two zero bits are inserted 'after' a number requiring at most 254 bits to represent. This guarantees that the result will be `Fr32`, regardless of the value of the initial 254 bits. This is a 'conservative' technique, since for some initial values, only one bit of zero-padding would actually be required. | ||
| - Steps 2 and 3 above are specific to the Lotus implementation. The same outcome can be achieved in different ways, e.g., without using `Fr32` bit-padding. However, any implementation has to make sure that the initial IPLD DAG is serialised and padded so that it gives a clean binary tree, and therefore, calculating the Merkle root out of the resulting blob of data gives the same **Piece CID**. As long as this is the case, implementations can deviate from the first three steps above. |
There was a problem hiding this comment.
Two things:
-
I do think the padding method is prescribed. Otherwise there will not be a deterministic mapping from raw data to CommP. It's just that this padding method is not the only possible way to convert from raw data into a sequence of
Fr32elements. -
I don't think anything about
Fr32is Lotus-specific: it's arust-fil-proofsimplementation detail.
I don't make these suggestions to complicate things. Probably incorporating them will lead to a simpler explanation. The current text is still not quite accurate.
There was a problem hiding this comment.
@porcuquine can you please provide the right phrasing for this, so that we finalise?
hugomrdias
changed the title
This PR is updating the description of the Filecoin Piece and provides a description of Data Representations in the Filecoin Network. Although the section will likely go through some restructuring, with this PR I would like to make sure that the content in this subsection is correct and up-to-date.