send() failed (32: Broken pipe) when performing HTTP POST to Immich

[Matthias-vdE]

When running Immich (https://github.com/immich-app/immich) behind NPM(plus) and enabling Crowdsec/Appsec, it is not possible to upload files to the server via HTTP POST:

2024-09-27T11:33:13.859502771Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 send() failed (32: Broken pipe), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859545312Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859553917Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859564759Z 2024/09/27 13:33:13 [alert] 32855#32855: *10529 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied 'MY_IPADDRESS' with 'ban' (by appsec), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

The issue was initially reported at ZoeyVid/NPMplus#1123.

[LaurenceJJones]

When running Immich (https://github.com/immich-app/immich) behind NPM(plus) and enabling Crowdsec/Appsec, it is not possible to upload files to the server via HTTP POST:

2024-09-27T11:33:13.859502771Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 send() failed (32: Broken pipe), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859545312Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859553917Z 2024/09/27 13:33:13 [error] 32855#32855: *10529 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

2024-09-27T11:33:13.859564759Z 2024/09/27 13:33:13 [alert] 32855#32855: *10529 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied 'MY_IPADDRESS' with 'ban' (by appsec), client: MY_IPADDRESS, server: immich.mydomain.org, request: "POST /api/assets HTTP/1.1", host: "immich.mydomain.org"

The issue was initially reported at ZoeyVid/NPMplus#1123.

Hey 👋🏻

I have setup a similar setup environment and don't see to be experiencing the same issue. I am uploading a 3mb file as per your reddit thread I dont see any errors.

Is CrowdSec / AppSec running locally to NPMPlus (on same host to reduce latency)?

Could you provide the full nginx configuration that is generated by NPMplus as I am using nginx (since its the same code I dont want to spend time configuring NPMPlus since its the same lua code)

server {
    server_name _;

    listen *:80;

    # allow large file uploads
    client_max_body_size 50000M;

    # Set headers
    proxy_set_header Host              $http_host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # enable websockets: http://nginx.org/en/docs/http/websocket.html
    proxy_http_version 1.1;
    proxy_set_header   Upgrade    $http_upgrade;
    proxy_set_header   Connection "upgrade";
    proxy_redirect     off;

    # set timeout
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;
    send_timeout       600s;

    location / {
        proxy_pass http://127.0.0.1:2283;
    }
}

everything seems to be getting processed:

root@bookworm:/etc/nginx/sites-enabled# cscli metrics show appsec
Appsec Metrics:
╭─────────────────┬───────────┬─────────╮
│ Appsec Engine   │ Processed │ Blocked │
├─────────────────┼───────────┼─────────┤
│ 127.0.0.1:7422/ │ 296       │ -       │
╰─────────────────┴───────────┴─────────

[Matthias-vdE]

Is CrowdSec / AppSec running locally to NPMPlus (on same host to reduce latency)?

Yup, both Crowdsec and NPMPlus are running in docker containers on the same host. Part of the same docker-compose:

services:
  npmplus:
    container_name: npmplus
    image: zoeyvid/npmplus:latest
    restart: always
    network_mode: host
    volumes:
      - "/opt/npm:/data"
    environment:
      - "TZ=Europe/Brussels" 
      - "NGINX_LOG_NOT_FOUND=true"
      - "LOGROTATE=true" 
      - "LOGROTATIONS=7" 
      - "GOA=true"

  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:latest
    restart: always
    network_mode: bridge
    ports:
      - "127.0.0.1:7422:7422"
      - "127.0.0.1:8080:8080"
    environment:
      - "TZ=Europe/Brussels"
      - "COLLECTIONS=ZoeyVid/npmplus"
      - "LEVEL_FATAL=true"
      - "LEVEL_ERROR=true"
      - "LEVEL_WARN=true"
      - "LEVEL_INFO=false"
      - "LEVEL_DEBUG=false"
      - "LEVEL_TRACE=false"
    volumes:
      - "/opt/crowdsec/conf:/etc/crowdsec"
      - "/opt/crowdsec/data:/var/lib/crowdsec/data"
      - "/opt/npm/nginx:/opt/npm/nginx:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  geoipupdate:
    container_name: geoipupdate
    image: maxmindinc/geoipupdate:latest
    restart: always
    network_mode: bridge
    environment:
      - "TZ=Europe/Brussels"
      - "GEOIPUPDATE_EDITION_IDS=GeoLite2-Country GeoLite2-City GeoLite2-ASN"
      - "GEOIPUPDATE_ACCOUNT_ID=my_account_id"
      - "GEOIPUPDATE_LICENSE_KEY=my_license_key"
      - "GEOIPUPDATE_FREQUENCY=24"
    volumes:
      - "/opt/npm/etc/goaccess/geoip:/usr/share/GeoIP"

Immich is running on the same host as well, as a docker container. The config for that one is very standard with little to no customization.

My proxy config in NPMPlus looks like this:

Nothing in custom locations, and a regular certbot certificate to force enable HTTPS.

My crowdsec.conf file is this:

ENABLED=true
API_URL=http://127.0.0.1:8080
API_KEY=my_API_key
CACHE_EXPIRATION=1
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=ban
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=live
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/data/etc/crowdsec/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
#valid providers are recaptcha, hcaptcha, turnstile
CAPTCHA_PROVIDER=
# Captcha Secret Key
SECRET_KEY=
# Captcha Site key
SITE_KEY=
CAPTCHA_TEMPLATE_PATH=/data/etc/crowdsec/captcha.html
CAPTCHA_EXPIRATION=3600
#APPSEC_URL=http://127.0.0.1:7422
#APPSEC_FAILURE_ACTION=deny

The two bottom lines are currently commented out to make it work. If I uncomment them, it breaks with the broken pipe error message when uploading anything.

[Zoey2936]

I don't use Immich myself, but there seems to be an Issue with it, I already have 3 discussions about it with multiple people having the same issues:
ZoeyVid/NPMplus#1168
ZoeyVid/NPMplus#1123
ZoeyVid/NPMplus#1241

the issue is always related to appsec and immich and disabling appsec (or changing APPSEC_FAILURE_ACTION to passthrough) fixed it (modsec also needs to be disabled) — sometimes increasing the timeouts also fixed (at least for Nextcloud which had similar issues), so I increased the default timeouts, but the issue still seems to exist on new installations with the new timeouts.

[MaximumFish]

To hopefully contribute, I started discussion 1241 that @Zoey2936 linked above. It was suggested I post here answering the below questions so here we go:

Is the deployment local or remote (VPS)? Local

Is the domain being proxied by a CDN like cloudflare? For full transparency I do use a Cloudflare tunnel, but only for external connections. The tunnel isn't involved while on the LAN.

Does it happen on upload like the OP or when you said when immich makes a backup? I am the OP in that thread. I believe the single image upload and backup feature are basically the same thing using the same endpoint. The backup feature just does it automatically and in batches.

This is what happens when I attempt to upload (or backup) a single photo:

2024/11/19 17:20:06 [warn] 80110#80110: *51203 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000050 while reading request body, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [error] 80110#80110: *51203 send() failed (32: Broken pipe), client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [error] 80110#80110: *51203 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [error] 80110#80110: *51203 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [alert] 80110#80110: *51203 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '10.42.0.11' with 'ban' (by appsec), client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"

(Domain name changed for privacy and yes, I use a non-typical subnet)

Disabling Appsec fixes the issue, albeit with the trade off of reduced security.

Thanks.

[blotus]

Hey,

Do you know how big is the body of the request when you get this error ?

Currently, the appsec will try to process any body it sees, regardless of the size, which will lead to issues (also tracked here: #71).

I haven't performed any real tests to see where the actual limit is currently, but I guess anything over a few hundred MBs will trigger this error.

Once #80 is merged (it includes a large refactoring of the code, so we have to wait for it), we plan to add additional configuration on how to handle large bodies (allow to set a maximum body size and whether to drop the request or just analyze the headers when it's over the limit).

[MFYDev]

I just met this issue today with my own Ghost blog post editing. appsec will block me from time to time as I was editing posts. No solution found yet but fortunately saw this issue, for post editing I think the body is not huge at all as I just started.

[MaximumFish]

Do you know how big is the body of the request when you get this error ?

The photos are a couple of MB each and it does one request per photo, so definitely not in the few hundred MB territory.

[yurividal]

Having the same issue.
I'm using nginxproxymanager with crowdsec_openresty_bouncer and appsec.

Issue happens even when inside the LAN, so no cloudflare on anything else involved.
Appsec is, in theory, setup properly and running fine:

curl -I -X POST localhost:7422/ -i -H 'x-crowdsec-appsec-api-key: {redacted}' -H 'x-crowdsec-appsec-ip: 42.42.42.42' -H 'x-crowdsec-appsec-uri: /test' -H 'x-crowdsec-appsec-host: test.com' -H 'x-crowdsec-appsec-verb: GET'   
HTTP/1.1 200 OK
Date: Tue, 03 Dec 2024 23:04:30 GMT
Content-Length: 36
Content-Type: text/plain; charset=utf-8

what could be causing the broken pipe between the appsec and the proxy?

edit:

Initially, the containers were on different docker networks, and communicating via ports exposed to the host.
I then moved the 2 containers to the same docker network. Now the error is a little different. connection reset by peer, instead of broken pipe.

error 10.10.10.20 -> photos.****.com : Allow(): AppSec check: connection reset by peer - POST /api/assets HTTP/1.1 - 
error 10.10.10.20 -> photos.****.com : AppSecCheck(): Fallback because of err: connection reset by peer - POST /api/assets HTTP/1.1 -