nextcloud-aio-nextcloud /tmp lacks size limit and can fill up crashing nextcloud and more

[k1n6b0b]

Steps to reproduce

1. Setup Nextcloud memories
2. index any file on external_storage larger then 50% of your free disk space occ memories:index
   Note: It appears this process copies the file from external_storage, to /tmp and then copies it again in /tmp using 2x the filesize while processing
3. sudo docker exec -it nextcloud-aio-nextcloud /bin/bash
4. watch 'ls -lha /tmp;

This file grows until it eventually fills my entire partition (~50GB)

54d900c19b86:/var/www/html# ls -lha /tmp/phpgABoHh 
-rw------- 1 www-data www-data 26G Oct  1 00:15 /tmp/phpgABoHh

Expected behavior

1. Better handling on whatever that tmp file is by the memories app! (See indexing large files on external_storage uses 2x filesize in /tmp and can crash nextcloud by consuming all available disk space pulsejet/memories#1305)
2. Better protections/limits on /tmp by nextlcoud-aio

Actual behavior

Filesystem fills up
database crashes
database container is left hanging in nextcloud network space (requiring system docker restart)
Upon restart, /tmp is cleared and everything works until the rescan fills the filesystem again :)

Indexing folder /XXXXX/files/nas_photoAlbums
PHP Notice:  fwrite(): Write of 331 bytes failed with errno=28 No space left on device in /var/www/html/lib/private/Log/File.php on line 87
Failed to index folder /XXXXX/files/nas_photoAlbums: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 7 no connection to the server

An exception occurred while executing a query: SQLSTATE[HY000]: General error: 7 no connection to the server

PHP Notice:  fwrite(): Write of 331 bytes failed with errno=28 No space left on device in /var/www/html/lib/private/Log/File.php on line 87

Error response from daemon: endpoint with name nextcloud-aio-database "already exists in network nextcloud-aio"

Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] connection to server at "nextcloud-aio-database" (172.18.0.5), port 5432 failed: Host is unreachable
	Is the server running on that host and accepting TCP/IP connections? in /var/www/html/lib/private/DB/Connection.php:167

~~Of note: After the 3rd crash and starting this report, I decided (4th run) to CTRL-C the memories:index with the file ~26GB and restarted it to see what it would do. The original 26GB tmp file was not cleaned up, but new tmp files are now being created/deleted and they are not growing unbounded. I deleted the 26GB tmp file and will let it run.

This may be hard to reproduce by the memories team -- but I still think some protections for the container's /tmp might be a good idea~~

Never mind, it crashed eventually in the same manner as before

Other information

Host OS

Ubuntu 24.04

Nextcloud Hub 8 (29.0.7)

Output of sudo docker info

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 13
  Running: 13
  Paused: 0
  Stopped: 0
 Images: 17
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: zfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: syslog
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-45-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.34GiB
 Name: nextcloud-aio
 ID: 106c73e1-6e35-44d3-bb0f-6b4b8d772ecf
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Docker run command or docker-compose file that you used

sudo docker run \
--init \
--sig-proxy=false \
--name nextcloud-aio-mastercontainer \
--restart always \
--publish 8080:8080 \
--env APACHE_PORT=11000 \
--env APACHE_IP_BINDING=0.0.0.0 \
--volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config \
--volume /var/run/docker.sock:/var/run/docker.sock:ro \
--env NEXTCLOUD_DATADIR="/mnt/ncdata" \
--env NEXTCLOUD_MEMORY_LIMIT=8192M \
--add-host nextcloud.XXX.XXX:10.XXX.XXX.XXX \
nextcloud/all-in-one:latest

Other valuable info

[szaimen]

Hi,

This file grows until it eventually fills my entire partition (~50GB)

So this seems to be the problem then in the end.

but I still think some protections for the container's /tmp might be a good idea

I fear I see no way for AIO to limit the size of the tmp directory. It could possibly be requested at https://github.com/nextcloud/server/issues though.

BTW, the tmp directory should in best case not fill up as hard as this. So this might be a bug in https://github.com/nextcloud/server

I only see this as a possiblity for you if your root drive is limited to 50GB: https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-to-store-the-filesinstallation-on-a-separate-drive

[k1n6b0b]

I can certainly expand the partition size I dedicated to /var/lib/docker, Is there guidance on what size this should be? All my data/db/etc is stored on volumes where I can control/anticipate/monitor size more easily. I do really feel there should be protections on tmp spaces so they dont crash containers this seems like a good practice for me. It took a while and multiple runs to determine what was happening

[szaimen]

I do really feel there should be protections on tmp spaces so they dont crash containers this seems like a good practice for me.

I agree in general but I don't see any way how to technically limit it inside AIO.

[szaimen]

I can certainly expand the partition size I dedicated to /var/lib/docker, Is there guidance on what size this should be?

Maybe 200 GB? However I agree that the tmp directory should not fill like that...

[JMarcosHP]

@szaimen We can do a better handling of /tmp partition, another issue I encounter is when you do public uploads.
For ex:

Many users can upload bigger files at the same time on a public link, so currently the nextloud-aio-nextcloud container can grow to much and fill up completely the partition where docker installation/files are.

Yo can test it by uploading a big file on a public link then, execute watch -n 1 'docker system df -v' until the upload completes, you will see when the container start to grow then move the uploaded file from /tmp/phpxxxxxxx in to the real location.

There are 2 solutions for this:

1. Create a request on https://github.com/nextcloud/server/issues to change the folder where public upload chunks are stored, for example the nextcloud data directory instead of /tmp (like when you upload files as a registered user data/user/uploads it can be data/public/uploads.

2. Set an env variable for Nextcloud-AIO to specify where to mount /tmp if the user has a fixed size partition or disk. NEXTCLOUD_TMP_DIR as the nextcloud documentation recommends. [1] [2]

[szaimen]

Lets track this in https://github.com/nextcloud/server/issues/48816

[ErikSteiner]

I have noticed the same behavior, with the "vanilla" Nextcloud docker image:
pulsejet/memories#1327

It filled up 42 GB of data under /tmp, before the host crashed.

[JMarcosHP]

Have you tried to set an external mount point for /tmp on vanilla nextcloud image using docker compose?

Unraid users have the same issue

linuxserver's image of nextcloud is more flexible, you can set a custom user and add more mountpoints. But lacks some other features.

The workaround for nextcloud aio is to move your entire docker installation to another disk with enough space, since the mantainers and nextcloud server devs have no plans to fix this.

Edit:
Reference

[ErikSteiner]

Thank you, for the suggesting. What can I expect from a bind mount for /tmp? I mean, how much space will be needed then?

What will happen, if I set /tmp as tmpfs? Will Nextcloud crash then, if the memory limit of the RAM is reached?

Furthermore, what I'm wondering is whether the folders like php10sO80 were created by uploading files or by generating the previews or memories? At least, these folders are located directly under /tmp.

$ ls -lah /tmp
total 43G
drwxrwxrwt 1 root root 4.0K Oct 29 14:35 .
drwxr-xr-x 1 root root 4.0K Oct 28 20:31 ..
-rwxr-xr-x 1 1004 1004 8.5M Oct 28 20:32 exiftool-ocgh0x2ccfv3-12.70-c435c63a
-rwxr-xr-x 1 1004 1004 7.4M Oct 28 20:36 exiftool-ocgh0x2ccfv3-12.70-f3319c53
-rwxr-xr-x 1 1004 1004 4.9M Oct 28 20:32 go-vod-ocgh0x2ccfv3-0.2.5-7345fee3
drwx------ 3 1004 1004 4.0K Oct 28 20:36 par-53595354454d
-rw------- 1 1004 1004 502M Oct 28 22:14 php0xPDBG
-rw------- 1 1004 1004 504M Oct 29 07:54 php10sO80
-rw------- 1 1004 1004 501M Oct 29 07:48 php1EO9WC
-rw------- 1 1004 1004 501M Oct 29 07:52 php2Oh3y6

[JMarcosHP]

Thank you, for the suggesting. What can I expect from a bind mount for /tmp? I mean, how much space will be needed then?

It depends on your Nextcloud usecase/workload/number of simultaneous users.
For example, I have a NAS and just want to see my files on Nextcloud, I have a dedicated disk to store all my containers and its data, my disk size is 320Gb and currently only 12Gb including nextcloud data folder are used, and my max file upload is 50Gb, I mount my other disk using the external storage app to avoid filling up my docker disk.

However, If you need to host Nextcloud for many users at the same time, then I recommend to have a dedicated disk just for temp file uploads, another for user data directory and a 3rd disk to store your docker installation/containers/backups. Size will depend on the number of users and hardware capacity/scalability.

What will happen, if I set /tmp as tmpfs? Will Nextcloud crash then, if the memory limit of the RAM is reached?

If you mount tmp on tmpfs you need to have enough ram capacity since tmpfs is made to store files on ram, if you run out ouf memory the system will start to use the swap, if you don't have a swapfile/partition it probably crash or kill some processes including docker/nextcloud.

I have written a guide to handle this use case here, I would like you to help me test and if it works, then I will make a pull request for the Nextcloud-AIO documentation or recommend this as a feature.