Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In version 5 of LiteDB, the file upload speed to the database has slowed down. #2544

Open
BibaAndBoba23 opened this issue Sep 18, 2024 · 4 comments
Labels

Comments

@BibaAndBoba23
Copy link

BibaAndBoba23 commented Sep 18, 2024

Good day! We are using the fourth version of LiteDB as a file cache. Upon upgrading to version 5, we noticed a significant drop in performance.
Example of data loading:

using LiteDatabase cache_database = new LiteDatabase(connectionString: this.connection_string); file_stream.Position = 0; LiteFileInfo lite_file_info = cache_database.FileStorage.Upload(id: $"{this.id_server}/{id_file}", filename: file_stream.Name, stream: file_stream);

maybe there are new ways to quickly download/upload files?

@BibaAndBoba23
Copy link
Author

You can see detailed information about the test results on this page:
http://bug.sdpss.ru/issue/SDP-13037/Nizkaya-skorost-zagruzki-fajlov-pri-obnovlenii-versii-LiteDB-do-5

@JKamsker
Copy link
Collaborator

The detailed information is hidden behind an auth wall.

@Lihoradit
Copy link

Hello, we've encountered an issue with LiteDB5 where the file upload speed is slower than in version 4. The screenshot below shows the measurements for a 1GB file.
image

@Lihoradit
Copy link

*Re-measurement with different file sizes:
Calculation: minutes/seconds

5 files 1Gb
Cache size 0Gb 11 Gb
LiteDb 5 17:41,84 20:50,11
LiteDB 4 05:14,66 11:00.91
File tree size 76 Mb
Cache size 0Gb 11Gb
LiteDB 5 00:12,20 00:14,45
LiteDB 4 00:07,50 00:08,72
Image 135Mb
Cache size 0Gb 11Gb
LiteDB 5 00:26,05 00:33,30
LiteDB 4 00:05,11 00:05,56
10 files 20 Mb
Cache size 0Gb 11Gb
LiteDb 5 00:43,02 00:45,94
LiteDB 4 00:12,34 00:10,76

-On LiteDB 4, there seems to be a cap after which nothing is written to the cache, even though there is 65GB free on the C drive.
-If you cancel the file copying process through document management, the temporary files remain in the cache folder and are not deleted.
-The cache write limit is always different; on 11.09, my cap was 9.7GB, and on 12.09, it was 11.8GB. This might be slowing down the file copying speed to the disk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants