Can't upload large files to Nextcloud (Chunks on server do not sum up)

[YeapGuy]

I get errors like this from Nextcloud clients when trying to upload large files:

Often, they sum up to 0 bytes, sometimes to an absurd number like here.

During the upload process, NPMplus behaves weirdly. I monitored memory usage and logs and attach them here. Maybe you know what's going on?

I can't figure out why this is happening... the only solution I came up with is switching to a different reverse proxy... I would really like to stay with NPMplus though...

Memory usage:

Logs:

2024/06/06 10:15:35 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000018 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00001 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:15:37 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000019 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000020 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:41 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:41 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:41 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:41 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:42 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000021 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:30 [error] 19486#19486: *35705 lua tcp socket read timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:30 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:30 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:30 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:32 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000022 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:25 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:25 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:25 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:25 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:27 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000023 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000024 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:43 [error] 19486#19486: *35705 lua tcp socket read timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:43 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:43 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:43 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:44 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000027 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:14 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:14 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:14 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:14 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:16 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000028 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00009 HTTP/1.1", host: "my-nextcloud.redacted.tld"

Nextcloud AIO proxy host custom config (according to their recommendation):

.env:

DISABLE_IPV6='true'
TZ='Europe/Bratislava'
LOGROTATE=true
GOA=false
GOACLA=--agent-list --real-os --double-decode --anonymize-ip --anonymize-level=2 --keep-last=7 --with-output-resolver --no-query-string

[Zoey2936]

Can you disable crowdsec and retry? (The advanced option are not needed in NPMplus, I wrote big parts of the npm/nginx docs of AIO)

[YeapGuy]

That helped. The memory jumps "only" to 1GB now (better than >2GB 😅), no errors are thrown and the upload finishes without any error. I want crowdsec though - why is this happening?

[Zoey2936]

Can you send me your crowdsec config?

[YeapGuy]

crowdsec.conf

ENABLED=false
API_URL=http://crowdsec:8080
API_KEY=REDACTED
CACHE_EXPIRATION=1
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=ban
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=live
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/data/etc/crowdsec/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
#valid providers are recaptcha, hcaptcha, turnstile
CAPTCHA_PROVIDER=hcaptcha
# Captcha Secret Key
SECRET_KEY=REDACTED
# Captcha Site key
SITE_KEY=REDACTED
CAPTCHA_TEMPLATE_PATH=/data/etc/crowdsec/captcha.html
CAPTCHA_EXPIRATION=3600
APPSEC_URL=http://crowdsec:7422
APPSEC_FAILURE_ACTION=deny

If you meant one of the crowdsec yaml files, please tell me which one, there's a lot of different configs for different things...

[Zoey2936]

Can you please remove those two (temporary) and test again:

APPSEC_URL=http://crowdsec:7422
APPSEC_FAILURE_ACTION=deny

appsec is a kind of WAF (similar to modsec), but only has rules which are unlikely to report a false positive, because of this I enable them by default. But since it is like a WAF every request is checked by crowdsec and I think that this could make problems with big files.

[YeapGuy]

Yup, it is the appsec component. With it disabled, it's working fine.

[Zoey2936]

Can you report that here: https://github.com/crowdsecurity/lua-cs-bouncer/issues