This repository has been archived by the owner on Aug 8, 2024. It is now read-only.
Jkl 330 implement the new file system #140
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…reaking them into blocks
…reduce memory load
…most at the end of the file
Old provider trims the buffer slice to the size of acutal bytes read from file. Creating merkle tree with trimmed vs untouched buffer results in different merkle tree. Thus will fail when posting proof to the chain as well. The issue is fixed by trimming the buffer to the size of actual bytes read from file.
…-330-implement-the-new-file-system
…-330-implement-the-new-file-system
TheMarstonConnell
approved these changes
Apr 15, 2024
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Storage server no longer breaks up files into chunks (0.jkl, 1.jkl ... n.jkl) and writes them into hard drive .
This improves cpu and memory usage especially on serving download requests as it no longer has to put all of split files back into its original form in memory before the file is sent to the user.
There are few optimizations on postproof and strays.
Migration
Updating requires migration from old file system to the new file system using
jprovd migratecommand.Per each file it'll: check merkle tree file, migrate the old file and check file integrity after migration.
If anything fails during the migration the file that failed will have its folder renamed with
corrupted-prefix and it won't be served. The migration command will print the details of the migration process so it's recommended to save it to some file.e.g.
jprovd migrate > result