fix/Child objects ACL #283
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| syntax = "proto3"; | ||
|
|
||
| package neo.fs.v2.link; | ||
|
|
||
| option go_package = "github.com/nspcc-dev/neofs-api-go/v2/link/grpc;link"; | ||
| option csharp_namespace = "Neo.FileStorage.API.Link"; | ||
|
|
||
| import "refs/types.proto"; | ||
|
|
||
| // Link is a payload of helper objects that contain the full list of the split | ||
| // chain objects' IDs. It is created only after the whole split chain is known | ||
| // and signed. This object is the only object that refers to every "child object" | ||
| // ID. It is NOT required for the original object assembling. It MUST have ALL | ||
| // the "child objects" IDs. Child objects MUST be ordered according to the | ||
| // original payload split, meaning the first payload part holder MUST be placed | ||
| // at the first place in the corresponding link object. Sizes MUST NOT be omitted | ||
| // and MUST be a real object payload size in bytes. | ||
| message Link { | ||
|
There was a problem hiding this comment. really deserves a separate package? to me its overhead. There was a problem hiding this comment. depends on the final solution about a separate object type. if it is a separate object type, i do not see any difference with There was a problem hiding this comment. imo the previously used approach only complicates the structuring and, as a consequence, the knowledge of the protocol concepts. All these entities are inextricably linked with objects - they are about objects, they are in the payload of objects. Сurrent packetization is extremely redundant to me
obviously, but we are not burdened with creating a new package (). Keep the existing ones, and this message can be left in being a newbie, i'll see There was a problem hiding this comment.
so do i. type of the objects are placed where they always have been the issue was created #284 There was a problem hiding this comment. 👍 for issue, |
||
| // Object ID with its object's payload size. | ||
| message MeasuredObject { | ||
| // Object ID. | ||
| neo.fs.v2.refs.ObjectID id = 1 [json_name = "id"]; | ||
|
|
||
| // Object size in bytes. | ||
| uint32 size = 2 [json_name = "size"]; | ||
| } | ||
|
|
||
| // Full list of the "child" object descriptors. | ||
| repeated MeasuredObject children = 1 [json_name = "children"]; | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| # Protocol Documentation | ||
| <a name="top"></a> | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| - [link/types.proto](#link/types.proto) | ||
|
|
||
| - Messages | ||
| - [Link](#neo.fs.v2.link.Link) | ||
| - [Link.MeasuredObject](#neo.fs.v2.link.Link.MeasuredObject) | ||
|
|
||
|
|
||
| - [Scalar Value Types](#scalar-value-types) | ||
|
|
||
|
|
||
|
|
||
| <a name="link/types.proto"></a> | ||
| <p align="right"><a href="#top">Top</a></p> | ||
|
|
||
| ## link/types.proto | ||
|
|
||
|
|
||
| <!-- end services --> | ||
|
|
||
|
|
||
| <a name="neo.fs.v2.link.Link"></a> | ||
|
|
||
| ### Message Link | ||
| Link is a payload of helper objects that contain the full list of the split | ||
| chain objects' IDs. It is created only after the whole split chain is known | ||
| and signed. This object is the only object that refers to every "child object" | ||
| ID. It is NOT required for the original object assembling. It MUST have ALL | ||
| the "child objects" IDs. Child objects MUST be ordered according to the | ||
| original payload split, meaning the first payload part holder MUST be placed | ||
| at the first place in the corresponding link object. Sizes MUST NOT be omitted | ||
| and MUST be a real object payload size in bytes. | ||
|
|
||
|
|
||
| | Field | Type | Label | Description | | ||
| | ----- | ---- | ----- | ----------- | | ||
| | children | [Link.MeasuredObject](#neo.fs.v2.link.Link.MeasuredObject) | repeated | Full list of the "child" object descriptors. | | ||
|
|
||
|
|
||
| <a name="neo.fs.v2.link.Link.MeasuredObject"></a> | ||
|
|
||
| ### Message Link.MeasuredObject | ||
| Object ID with its object's payload size. | ||
|
|
||
|
|
||
| | Field | Type | Label | Description | | ||
| | ----- | ---- | ----- | ----------- | | ||
| | id | [neo.fs.v2.refs.ObjectID](#neo.fs.v2.refs.ObjectID) | | Object ID. | | ||
| | size | [uint32](#uint32) | | Object size in bytes. | | ||
|
|
||
| <!-- end messages --> | ||
|
|
||
| <!-- end enums --> | ||
|
|
||
|
|
||
|
|
||
| ## Scalar Value Types | ||
|
|
||
| | .proto Type | Notes | C++ Type | Java Type | Python Type | | ||
| | ----------- | ----- | -------- | --------- | ----------- | | ||
| | <a name="double" /> double | | double | double | float | | ||
| | <a name="float" /> float | | float | float | float | | ||
| | <a name="int32" /> int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | | ||
| | <a name="int64" /> int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | | ||
| | <a name="uint32" /> uint32 | Uses variable-length encoding. | uint32 | int | int/long | | ||
| | <a name="uint64" /> uint64 | Uses variable-length encoding. | uint64 | long | int/long | | ||
| | <a name="sint32" /> sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | | ||
| | <a name="sint64" /> sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | | ||
| | <a name="fixed32" /> fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | | ||
| | <a name="fixed64" /> fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | | ||
| | <a name="sfixed32" /> sfixed32 | Always four bytes. | int32 | int | int | | ||
| | <a name="sfixed64" /> sfixed64 | Always eight bytes. | int64 | long | int/long | | ||
| | <a name="bool" /> bool | | bool | boolean | boolean | | ||
| | <a name="string" /> string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | | ||
| | <a name="bytes" /> bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if we wanna use it, we should be able to rely on it so there is MUST word. but then, we need to validate it and that is N head requests per link object PUT (N is num of objects in the chain)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the problem is more specific - when assembling a split object, the list of descriptors of child objects can diverge with the stored children (not only the sizes btw, IDs too). The proposed validation is correct, but since we cannot guarantee it at the protocol level (this is a server implementation detail), the assembler - one who operates with split chain - must be prepared for the divergence in any case
protocol-level solution is to additionally store such metadata in the child object headers themselves (mentioned here). Then the link object will remain a pure stand-alone helper
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what do you mean? nodes may not accept such link objects
there is no solution for #264 then. if you cannot trust the link object, you cannot be sure you are answering with a correct range
should also be validated somehow. a node can get Nth object with some offset in it. what does it do? just believes it is true? searches for the previous ones and calculates their sizes?
@roman-khimov
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nodes may do whatever they want. They may accept linking objects w/o any validation
i dont deny current solution, im just highligting that linking object can be unavailable or diverge with the split chain and any network node must be ready for this. In these cases, extra metadata in child objects may be a painkiller in some cases
its own implementation detail, each node can only control what to do itself and what it receives from other nodes
or maybe i misunderstood ur original comment. I dont see any problem with current structure and requirements. If u didnt mean that
N head requests per link object PUTis a problem, then everything is ok to meThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i meant we need to find a balance to implement our "perfect" node that may exist in the world of "worst" nodes. if we may not rely on link object, i think the whole work is useless. but if we want to rely on it, we should validate it somehow
here i want to say that ensuring child object data is ok is the same to ensuring similar info about the link object
yes. this and the consequences
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Validation should be implemented. Then I'd be optimistic wrt the link object contents. If it's not correct --- either you get an error (trying to access inexisting object) or reply with a wrong result, but it's not node's fault, garbage in -- garbage out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, sure. Just not sure about some link object replication: imagine you have a 1к parts object and receive a link to it for the first time: you need to do 1k head objects before you finish with this object. And you may be an attacker that sends 1k of such 1k-parts objects. 64mb data for a HEAD chaos in the NeoFS network.