Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement/lp refactor download #753

Merged
merged 56 commits into from
May 1, 2023
Merged

Conversation

lpoli
Copy link
Contributor

@lpoli lpoli commented Feb 6, 2023

Changes

Following changes are added:

  1. Add reader that complies with io.ReadSeekCloser interface
  2. Provide download function for zboxcli to read file into some file; it also serves as an example to how to use reader

Fixes

Tests

Tasks to complete before merging PR:

  • Ensure system tests are passing. If not Run them manually to check for any regressions 📋
  • Do any new system tests need added to test this change? do any existing system tests need updated? If so create a PR at 0chain/system_test
  • Merge your system tests PR to master AFTER merging this PR

Associated PRs (Link as appropriate):

Vertical can have multiple blocks per read marker

Horizontal can have only one block per read mearker
Define more errors
Refactor StreamDownld structure
Get refs with path hash and authticket as well
Few fixes and optimization
Remove print statements
@lpoli lpoli linked an issue Feb 6, 2023 that may be closed by this pull request
@dabasov dabasov marked this pull request as ready for review March 22, 2023 14:11
@lpoli lpoli mentioned this pull request Apr 8, 2023
3 tasks
return 0, err
}

offset := sd.offset % int64(effectiveChunkSize)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what does the mod do here. And seems it could cause data lose. Imagine we request block_1, the offset is 109, and effectiveChunkSize is 100, then we will copy the data[9:] in block_1 to b. So where's the data[:9] going.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peterlimg If client requests with offset 109, then it means client does not want or care about data[:9]. So for example, if client already had stored 109 bytes of data then for requesting other data offset would be set to 109. Since data is requested in block level(100 bytes in above example), 109 bytes means client has stored 100 bytes from block_0 and 9 bytes from block_1, so now after getting block_1, it should skip data[:9] otherwise data will be corrupted.

@lpoli lpoli requested a review from peterlimg April 27, 2023 02:22
Copy link
Member

@peterlimg peterlimg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@peterlimg peterlimg merged commit 0885b6e into staging May 1, 2023
@peterlimg peterlimg deleted the enhancement/lp-refactor-download branch May 1, 2023 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

refactor download file
3 participants