Blog post on UseGalaxy.eu's rustus migration

[kysrpex]

A blog post on the recent migration from tusd to rustus on UseGalaxy.eu and the reasons behind.

The blog post is still a draft solely because we should find a good source that explains why storing tens of thousands of files in the same directory is disadvantageous.

Given the high volume of uploads on our server, this is advantageous for scaling our service, because we can avoid storing tens of thousands of files in the same directory.

A table that compares the features of tusd and rustus has also been included in the PR even though it will not be displayed in the blog post (just for our own reference).

[bgruening]

A table that compares the features of tusd and rustus has also been included in the PR even though it will not be displayed in the blog post (just for our own reference).

I guess there is nothing wrong to put this into the blog post as well.

[kysrpex]

I actually do not see the reason why having hundreds of thousands of files is inherently bad. I have been checking different sources and found nothing that backs up this general claim.

As far as I understood, old filesystems had this problem but modern filesystems use B or B+ trees [1], so this becomes a non-issue. There are even people that have benchmarked flat and deep directory structures [2] and the outcome is that, given that you know the path of the file to be read and/or written, the deep directory structure is even counterproductive.

I can understand that for different applications that are not exclusively simple read or write operations, like listing the contents of a directory, removing old uploads, and others definitely there is an advantage in having the directory structure.

I lack the knowledge of how we are cleaning up the old uploads at the moment, so I can neither claim that the deep directory structure an advantage nor a disadvantage for us. Information is welcome.

[1] - What are the performance implications for millions of files in a modern file system? - https://serverfault.com/a/796696
[2] - Benchmark: Deep directory structure vs. flat directory structure to store millions of files on ext4 - https://medium.com/@hartator/benchmark-deep-directory-structure-vs-flat-directory-structure-to-store-millions-of-files-on-ext4-cac1000ca28

[kysrpex]

Information is welcome.

Thanks @bgruening and @sj213 for the help!

[bgruening]

Cool, thanks.