Switch to v2 object split scheme #957

smallhive · 2024-05-28T11:12:58Z

No description provided.

go.mod

cmd/s3-gw/app_settings.go

internal/neofs/tree.go

api/layer/tree_service.go

api/data/tree.go

api/layer/multipart_upload.go

api/layer/neofs.go

api/layer/neofs_mock.go

api/layer/tree_mock.go

api/layer/tree_service.go

api/layer/neofs_mock.go

codecov · 2024-06-05T09:14:12Z

Codecov Report

Attention: Patch coverage is 9.13462% with 189 lines in your changes missing coverage. Please review.

Project coverage is 26.71%. Comparing base (d6d4636) to head (e016f43).
Report is 32 commits behind head on master.

❗ Current head e016f43 differs from pull request most recent head 8328780

Please upload reports for the commit 8328780 to get more accurate results.

Files	Patch %	Lines
api/layer/multipart_upload.go	0.00%	71 Missing ⚠️
api/layer/neofs_mock.go	0.00%	36 Missing and 1 partial ⚠️
internal/neofs/tree.go	0.00%	30 Missing ⚠️
api/data/tree.go	0.00%	16 Missing ⚠️
internal/neofs/neofs.go	0.00%	13 Missing ⚠️
api/layer/tree_mock.go	0.00%	9 Missing ⚠️
api/handler/object_list.go	12.50%	6 Missing and 1 partial ⚠️
api/handler/multipart_upload.go	0.00%	4 Missing ⚠️
cmd/s3-gw/app_settings.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #957      +/-   ##
==========================================
- Coverage   26.92%   26.71%   -0.21%     
==========================================
  Files          87       87              
  Lines       14129    14246     +117     
==========================================
+ Hits         3804     3806       +2     
- Misses       9903    10017     +114     
- Partials      422      423       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

api/handler/multipart_upload.go

internal/neofs/tree.go

api/handler/object_list.go

internal/neofs/tree.go

roman-khimov

I've expected more red lines, but probably we're still too dependent on the tree service.

api/handler/multipart_upload.go

roman-khimov · 2024-06-13T15:04:17Z

api/layer/multipart_upload.go

+	var (
+		bktInfo         = p.Bkt
+		attributes      [][2]string
+		multipartHash   = sha256.New()


Technically it's not needed, you know it has no data.

It is correct. Unfortunately, I failed to avoid it. Even empty data must be hashed and marshaled to be available for the next parts. This required calculating a hash for the whole object and the next part uses the "hash state" from the previous one

Closes #937. Signed-off-by: Evgenii Baidakov <[email protected]>

In real NeoFS the complex object is constructing by the node using linking object, previous-id header, etc. For tests we use only the map and this step creates real object in this map. Signed-off-by: Evgenii Baidakov <[email protected]>

api/handler/object_list.go

api/layer/tree_service.go

These changes fixes test_s3.py::test_versioning_obj_suspend_versions and test_s3.py::test_list_multipart_upload_owner tests. Signed-off-by: Evgenii Baidakov <[email protected]>

Fixes s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_multipart_bad_download test. Signed-off-by: Evgenii Baidakov <[email protected]>

Signed-off-by: Evgenii Baidakov <[email protected]>

Current texts are misleading a bit. Signed-off-by: Evgenii Baidakov <[email protected]>

Signed-off-by: Evgenii Baidakov <[email protected]>

carpawell

This is (has been for 2 weeks) my first experience with our S3 review. Overall, looks working to me. However, I still do not understand why some changes are done now and in the replace v1 to v2 commit. We have not changed the split noticeably, it is just now more convenient and more informative for SNs, but it is still an object chain, and still, the whole chain is backward-dependent, so it is strange.

Also, reuploading parts logic blows my mind, but it is not this PR's scope.

smallhive requested a review from roman-khimov as a code owner May 28, 2024 11:12

smallhive mentioned this pull request May 28, 2024

GET of a multipart object fails #954

Closed

smallhive linked an issue May 28, 2024 that may be closed by this pull request

GET of a multipart object fails #954

Closed

roman-khimov requested a review from carpawell May 29, 2024 11:55

roman-khimov reviewed May 29, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

cmd/s3-gw/app_settings.go Outdated Show resolved Hide resolved

internal/neofs/tree.go Show resolved Hide resolved

api/layer/tree_service.go Show resolved Hide resolved

smallhive force-pushed the 937-switch-to-v2-object-split-scheme branch from f242f22 to 0bff48e Compare May 30, 2024 07:04

carpawell reviewed May 30, 2024

View reviewed changes

smallhive force-pushed the 937-switch-to-v2-object-split-scheme branch from 0bff48e to 54a811f Compare May 31, 2024 05:09

smallhive requested review from roman-khimov and carpawell June 3, 2024 09:46

smallhive linked an issue Jun 10, 2024 that may be closed by this pull request

Switch to v2 object split scheme #937

Closed

roman-khimov reviewed Jun 10, 2024

View reviewed changes

smallhive force-pushed the 937-switch-to-v2-object-split-scheme branch 3 times, most recently from b7c4cba to cf43c02 Compare June 11, 2024 04:27

smallhive requested a review from roman-khimov June 11, 2024 04:30

smallhive mentioned this pull request Jun 13, 2024

Incorrect owner id in list object versions output #953

Closed

smallhive force-pushed the 937-switch-to-v2-object-split-scheme branch from cf43c02 to c699af0 Compare June 13, 2024 13:40

roman-khimov reviewed Jun 13, 2024

View reviewed changes

smallhive added 2 commits June 14, 2024 11:34

*: Replace split V1 to V2

72e8b8c

Closes #937. Signed-off-by: Evgenii Baidakov <[email protected]>

layer: Fix internal tests

850d9d5

In real NeoFS the complex object is constructing by the node using linking object, previous-id header, etc. For tests we use only the map and this step creates real object in this map. Signed-off-by: Evgenii Baidakov <[email protected]>

smallhive force-pushed the 937-switch-to-v2-object-split-scheme branch from c699af0 to 2931301 Compare June 14, 2024 08:09

smallhive requested a review from roman-khimov June 14, 2024 08:10

roman-khimov reviewed Jun 14, 2024

View reviewed changes

api/handler/object_list.go Outdated Show resolved Hide resolved

api/layer/tree_service.go Show resolved Hide resolved

smallhive added 5 commits June 14, 2024 15:58

*: Return correct info about owner

b69688c

These changes fixes test_s3.py::test_versioning_obj_suspend_versions and test_s3.py::test_list_multipart_upload_owner tests. Signed-off-by: Evgenii Baidakov <[email protected]>

handler: Return real object size from header for encrypted objects.

cb41b6e

Fixes s3tests_boto3/functional/test_s3.py::test_encryption_sse_c_multipart_bad_download test. Signed-off-by: Evgenii Baidakov <[email protected]>

workflow: Update component versions

aff34aa

Signed-off-by: Evgenii Baidakov <[email protected]>

neofs: Update error messages

b5f56f8

Current texts are misleading a bit. Signed-off-by: Evgenii Baidakov <[email protected]>

*: Add zero part and use it as multipart uploadID

8328780

Signed-off-by: Evgenii Baidakov <[email protected]>

smallhive force-pushed the 937-switch-to-v2-object-split-scheme branch from 2931301 to 8328780 Compare June 14, 2024 11:59

roman-khimov approved these changes Jun 14, 2024

View reviewed changes

carpawell approved these changes Jun 14, 2024

View reviewed changes

roman-khimov merged commit 0c5bdbe into master Jun 14, 2024
16 of 17 checks passed

roman-khimov deleted the 937-switch-to-v2-object-split-scheme branch June 14, 2024 15:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to v2 object split scheme #957

Switch to v2 object split scheme #957

smallhive commented May 28, 2024

codecov bot commented Jun 5, 2024 •

edited

Loading

roman-khimov left a comment

roman-khimov Jun 13, 2024

smallhive Jun 14, 2024

carpawell left a comment •

edited

Loading

Switch to v2 object split scheme #957

Switch to v2 object split scheme #957

Conversation

smallhive commented May 28, 2024

codecov bot commented Jun 5, 2024 • edited Loading

Codecov Report

roman-khimov left a comment

Choose a reason for hiding this comment

roman-khimov Jun 13, 2024

Choose a reason for hiding this comment

smallhive Jun 14, 2024

Choose a reason for hiding this comment

carpawell left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Jun 5, 2024 •

edited

Loading

carpawell left a comment •

edited

Loading