【filesystem as backend】the filesystem is not local storage, it is like NFS. When uploading file is going to finish, it is stuck.

[stephenix]

Hi, @gaul

Describe the problem

I use NFS as my backend filesystem, but I need s3 API. So I use s3proxy+NFS.

However, when I upload a file like 1GB through s3proxy, the file is going to be uploaded completely, I must wait a long time for it to end.

I open the s3proxy debug log，find many Retrieve/Delete log:

[s3proxy] D 09-18 03:40:30.650 Thread-37 o.j.b.config.LocalBlobStore:56 |::] Retrieving blob with key .mpus-bfef012a-d576-45f9-951c-f8e7c6332760-10-2 from container upload
[s3proxy] D 09-18 03:40:30.650 Thread-37 o.j.b.config.LocalBlobStore:56 |::] Opening blob in container: upload - .mpus-bfef012a-d576-45f9-951c-f8e7c6332760-10-2
[s3proxy] D 09-18 03:40:30.915 Thread-37 o.j.b.config.LocalBlobStore:56 |::] Put blob with key [10] to container [upload]
[s3proxy] D 09-18 03:40:30.943 Thread-37 o.j.b.config.LocalBlobStore:56 |::] Retrieving blob with key .mpus-bfef012a-d576-45f9-951c-f8e7c6332760-10-1 from container upload
[s3proxy] D 09-18 03:40:30.943 Thread-37 o.j.b.config.LocalBlobStore:56 |::] Opening blob in container: upload - .mpus-bfef012a-d576-45f9-951c-f8e7c6332760-10-1
[s3proxy] D 09-18 03:40:32.985 Thread-37 o.j.b.config.LocalBlobStore:56 |::] Retrieving blob with key .mpus-bfef012a-d576-45f9-951c-f8e7c6332760-10-2 from container upload
[s3proxy] D 09-18 03:40:33.033 Thread-37 o.j.b.config.LocalBlobStore:56 |::] Opening blob in container: upload - .mpus-bfef012a-d576-45f9-951c-f8e7c6332760-10-2
[s3proxy] D 09-18 03:40:33.882 Thread-37 o.j.f.s.i.FilesystemStorageStrategyImpl:56 |::] Deleting blob /mnt/upload/upload/.mpus-bfef012a-d576-45f9-951c-f8e7c6332760-10-1
[s3proxy] D 09-18 03:40:33.909 Thread-37 o.j.f.s.i.FilesystemStorageStrategyImpl:56 |::] Deleting blob /mnt/upload/upload/.mpus-bfef012a-d576-45f9-951c-f8e7c6332760-10-2
[s3proxy] D 09-18 03:40:33.936 Thread-37 o.j.f.s.i.FilesystemStorageStrategyImpl:56 |::] Deleting blob /mnt/upload/upload/.mpus-bfef012a-d576-45f9-951c-f8e7c6332760-10-stub
[s3proxy] D 09-18 03:40:33.990 S3Proxy-Jetty-67 o.j.f.s.i.FilesystemStorageStrategyImpl:56 |::] Deleting blob /mnt/upload/upload/bfef012a-d576-45f9-951c-f8e7c6332760

I see many temporary parts file .mpus-xxxxxx, 8MB, the count is 1000GB/8MB=125
I can understand 1GB file is split into temporary multi part, after upload, the parts should be deleted, so it is like stuck.

Expected behavior

Can I add conf in s3proxy.conf to make temporary multi part in memory or local disk?
If this works, it will significantly reduce the wait time.

Or increase the 8MB size?