Cannot upload large files when using chunking

[AndreiArdei]

Hi! I've recently set up nextcloud using docker. The OS that docker runs on is Ubuntu 22.04.4 LTS. I provided the docker compose file below. My docker volume is mounted on a sepparate NFS share, therefore all volumes are mounted on the nfs share.

version: '3.8'

services:
  postgres:
    image: postgres:alpine
    container_name: nextcloud-postgres
    restart: unless-stopped
    environment:
      - POSTGRES_PASSWORD=nextcloud
      - POSTGRES_DB=nextcloud
      - POSTGRES_USER=nextcloud
    volumes:
      - nextcloud_db:/var/lib/postgresql/data

  redis:
    image: redis:alpine
    container_name: nextcloud-redis
    restart: unless-stopped

  app:
    image: nextcloud:apache
    restart: unless-stopped
    container_name: nextcloud
    volumes:
      - nextcloud_app_data:/var/www/html/data
      - nextcloud_app_opt:/var/www/html
    environment:
      - POSTGRES_HOST=postgres
      - REDIS_HOST=redis
      - POSTGRES_PASSWORD=nextcloud
      - POSTGRES_DB=nextcloud
      - POSTGRES_USER=nextcloud
      - PHP_UPLOAD_LIMIT=20G
      - APACHE_BODY_LIMIT=0
    depends_on:
      - postgres
      - redis
    ports:
      - 7575:80

volumes:
  nextcloud_db:
  nextcloud_app_data:
  nextcloud_app_opt:

The problem I run into is that when I try to upload larger files, this fails with the error:
[PHP] Error: rmdir(/mnt/ncdata/admin/uploads/web-file-upload-15252b578fdf95d6740fb72e4e6c488a-1687753536511): Directory not empty at /var/www/html/lib/private/Files/Storage/Local.php#147

I managed to find the following link to problem however, there it is specified that the error is in the alpine version, where as I am using the nextcloud:apache tag. I was able to have a workaround by turning off chunking with the following command:

sudo -u www-data php occ config:app:set files max_chunk_size --value 0

This seems to allow larger files now, however it has slowed down upload time considerably. Is there any known fix or work-around / other image that I could use to work around this problem?

[tzerber]

Are you uploading the files via web or NC client ? I just finished uploading 38 gig file and had 0 issues. Over the last two years I've uploaded terabytes of data via the NC app and had only one issue, and that was a datacenter outage that caused some weird database issues, but after fixing it I still have no problems with large files. Max i've tried is approx 500 gigabytes file and it went okay.

[AndreiArdei]

Unfortunately with both the web and the NC client. The NC client seems to be a tad faster than over the web but still slow. IT does work if i turn of chunking (i.e. set max_chunk_size) to 0 but then the upload speed is next to nothing for a 1.5GB file. If I set it to the default 10MB or even 20MB the upload speed seems to pick up but it fails.

[tzerber]

Using web for large (>5GB) files is always pain due php limits. The NC desktop client should not have any issues with chunked upload. Can you try to revert the change to max-chunk-size and try with the desktop app ?

[AndreiArdei]

Just for my peace of mind, this is also how I attach the NFS share.

192.168.1.139:/mnt/pool01/docker /var/lib/docker/volumes nfs x-systemd.automount,rw,soft,sync 0 0

Basically I mount the entire docker volumes directory to the nfs share. Meaning all volumes are actually created on the share not on the machine where I run docker (and nextcloud) from.

I did two tests with the NC Client:

1. set max_chunk_size to 0 -> Completed the upload though it took a very long time
2. set max_chunk_size to 20MB -> Failed (I got multiple errors but can't find them anymore)

I am now retrying with the 20MB chunking from the desktop and then I will also set it to the default 10MB chunking and try again.

[AndreiArdei]

Just tried with 10MB chunking size. Still fails:

This time however I don't get any sort of errors in the server logs.

[joshtrichards]

[PHP] Error: rmdir(/mnt/ncdata/admin/uploads/web-file-upload-15252b578fdf95d6740fb72e4e6c488a-1687753536511): Directory not empty at /var/www/html/lib/private/Files/Storage/Local.php#147 

What is /mnt/ncdata? That's not indicated in your provided Compose.

[AndreiArdei]

[PHP] Error: rmdir(/mnt/ncdata/admin/uploads/web-file-upload-15252b578fdf95d6740fb72e4e6c488a-1687753536511): Directory not empty at /var/www/html/lib/private/Files/Storage/Local.php#147 

What is /mnt/ncdata? That's not indicated in your provided Compose.

That is a really good question and i am not sure. I've looke both in the container, on the machine running docker and on the machine that hosts the volumes, it doesn't exist ? I can't find it anywhere. Als it's not something i'd setup since it doesn't follow my format for naming volumes or anyhting else.

https://help.nextcloud.com/t/what-is-mnt-ncdata-for/189789
Edit: I did come across this, however I still cannot find it on the host machine as a real directory.

[joshtrichards]

Did you previously use the All-In-One image? I believe it's the default datadirectory for that image.

Is it referenced under occ config:list system --private | grep ncdata? Or perhaps are there some symbolic links in your data directory?

[AndreiArdei]

I did indeed try an all-in-one image months ago, however that did not work for me so I removed the container, image and volumes related to it. Running the command you provided results in no data. Even if i don't do the grep for ncdata, this directory isn't there anyway.

Also cannot seem to find any symbolic links to it

[joshtrichards]

The way this image works and based on your Compose:

/mnt/ncdata/admin/uploads/web-file-upload-15252b578fdf95d6740fb72e4e6c488a-1687753536511

should be:

/var/www/html/data/admin/web-file-upload-15252b578fdf95d6740fb72e4e6c488a-1687753536511

Do you have any data in your user's home directory within the container - e.g. /var/www/html/data/admin should contain files you've created or uploaded?