Superfluous nginx deployment?

[JosefWN]

Looking to potentially switch to the FPM image. If I understood things correctly, using the fpm image, you would have a request/response flow like this:

nextcloud-fpm-alpine <-> nginx <-> ingress-nginx

To my understanding, this sort of defeats the purpose of using FastCGI to reduce latencies/bandwidth in the first place?

Is there any reason this cannot be simplified to:

nextcloud-fpm-alpine <-> ingress-nginx

FastCGI support in ingress-nginx: https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services/

Am I missing something?

[provokateurin]

Sounds interesting! I think nobody thought about it. Would be interested to test this out if you make a PR.

[provokateurin]

I guess this depends in the ingress you are using, but we should definitely have an option for enabling it.

[JosefWN]

I think the major ingress controllers like ingress-gce, ingress-nginx, Traefik etc. support it, but I'm not sure if some special care needs to be taken w.r.t static content. Not really experienced in this. If I have time I'll have a look, currently using a home-rolled chart for Nextcloud and came here for inspiration, but I will share my findings if I come up with anything!

[tcoupin]

I'm interresing into this enhancement.

What do you think about remove the nginx part and modify the image key such as:

image:
  repository: nextcloud
  # version: "24.0.3" 
  type: "apache"
  # tag: 24.0.3-apache
  pullPolicy: IfNotPresent
  # pullSecrets:
  #   - myRegistrKeySecretName

The user can set the image tag by image.tag or the tag will be determine by concatenate image.version and type.

At the ingress section, add a vendor key to specify ingress-nginx, traefik etc. Depending on this, the suitables annotations will be add to the ingress (if image.type is fpm, of course...)

[tcoupin]

The probes parts must be updated too. When image.type is fpm, livenessprobe must execute occ status command.

Currently, the probes are disabled :

{{- if and .Values.livenessProbe.enabled (not .Values.nginx.enabled) }}
        livenessProbe:

[jessebot]

What do you think about remove the nginx part and modify the image key such as:

image:
 repository: nextcloud
 # version: "24.0.3" 
 type: "apache"
 # tag: 24.0.3-apache
 pullPolicy: IfNotPresent
 # pullSecrets:
 #   - myRegistrKeySecretName

The user can set the image tag by image.tag or the tag will be determine by concatenate image.version and type.

@tcoupin we currently have image.flavor which would be the same as image.type.

I experimented with this a little bit after #386 was merged, and I could set the nextcloud.containerPort to 9000, and didn't have much luck. I kept getting an error on nextcloud.dev.coolwebsitefordogs.com/index.php/login, but I forgot to take a screenshot. Will try to post my findings here. Currently testing on GKE and K3s with the ingress-nginx controller.

Here's the config map I used:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nextcloud-fast-cm
  namespace: nextcloud
data:
  SCRIPT_FILENAME: "/var/www/html/index.php"
  PATH_INFO: "$path_info"
  HTTPS: "on"
  modHeadersAvailable: true
  # Enable pretty urls
  front_controller_active: true

I can't figure out what parameters to pass into the ConfigMap to make this work 🤔

Here's my image and ingress parts of my values.yaml:

image:
  repository: nextcloud
  # image will be nextcloud:$APP_VERSION-fpm e.g. nextcloud:26.0.1-fpm
  flavor: fpm

ingress:
  enabled: true
  # className: ingress-nginx
  annotations:
    kubernetes.io/ingress.class : "ingress-nginx"
    cert-manager.io/cluster-issuer: "letsencrypt"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "FCGI"
    nginx.ingress.kubernetes.io/fastcgi-index: "index.php"
    nginx.ingress.kubernetes.io/fastcgi-params-configmap: "nextcloud-fast-cm"
    nginx.ingress.kubernetes.io/proxy-body-size: 4G
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For"
    nginx.ingress.kubernetes.io/server-snippet: |-
      server_tokens off;
      proxy_hide_header X-Powered-By;

      rewrite ^/.well-known/webfinger /public.php?service=webfinger last;
      rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
      rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;
      location = /.well-known/carddav {
        return 301 $scheme://$host/remote.php/dav;
      }
      location = /.well-known/caldav {
        return 301 $scheme://$host/remote.php/dav;
      }
      location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
      }
      location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
        deny all;
      }
      location ~ ^/(?:autotest|occ|issue|indie|db_|console) {
        deny all;
      }
  tls:
    - secretName: nextcloud-tls
      hosts:
        - nextcloud.dev.coolwebsitefordogs.com
  labels: {}
  path: /

[jessebot]

@JosefWN were you ever able to experiment with this?

Also open to anyone else in the community that may have had experience with this, as I'm unsure how to make this work.

[jessebot]

I still haven't had success in making this work, but I got a lot further than last time. Here's where I got to the other day while testing this again:

I used the the branch in this PR to test everything, because I needed to be able to set securityContext without having an nginx container: #379

Here's my FastCGI configMap for the ingress-nginx object to use that I deployed to the nextcloud namespace on my cluster:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
  name: nextcloud-fastcgi-cm
data:
  modHeadersAvailable: "true"
  front_controller_active: "true"
  DOCUMENT_ROOT: "/var/www/html"
  SCRIPT_FILENAME: "$document_root$fastcgi_script_name"
  PATH_INFO: "$fastcgi_path_info"
  HTTPS: "1"

Image and ingress section of my values.yaml:

image:
  flavor: fpm-alpine
  
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod

    nginx.ingress.kubernetes.io/backend-protocol: "FCGI"
    nginx.ingress.kubernetes.io/fastcgi-index: "index.php"
    nginx.ingress.kubernetes.io/fastcgi-params-configmap: "nextcloud-fastcgi-cm"

    nginx.ingress.kubernetes.io/proxy-body-size: 10G

    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For"
    nginx.ingress.kubernetes.io/server-snippet: |-
      location ~ \.php(?:$|/) {
        fastcgi_split_path_info ^(.+?\.php)(/.*)$;
        set $path_info $fastcgi_path_info;
      }

      proxy_hide_header X-Powered-By;
      rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last;
      rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last;
      rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
      rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;

      location = /.well-known/carddav {
        return 301 $scheme://$host/remote.php/dav/;
      }

      location = /.well-known/caldav {
        return 301 $scheme://$host/remote.php/dav/;
      }

  tls:
    - secretName: nextcloud-tls
      hosts:
        - "cloud.example.com"

click me for the full values.yaml

image:
  flavor: fpm-alpine

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod

    nginx.ingress.kubernetes.io/backend-protocol: "FCGI"
    nginx.ingress.kubernetes.io/fastcgi-index: "index.php"
    nginx.ingress.kubernetes.io/fastcgi-params-configmap: "nextcloud-fastcgi-cm"

    nginx.ingress.kubernetes.io/proxy-body-size: 10G

    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For"
    nginx.ingress.kubernetes.io/server-snippet: |-
      location ~ \.php(?:$|/) {
        fastcgi_split_path_info ^(.+?\.php)(/.*)$;
        set $path_info $fastcgi_path_info;
      }

      proxy_hide_header X-Powered-By;
      rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last;
      rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last;
      rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
      rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;

      location = /.well-known/carddav {
        return 301 $scheme://$host/remote.php/dav/;
      }

      location = /.well-known/caldav {
        return 301 $scheme://$host/remote.php/dav/;
      }

  tls:
    - secretName: nextcloud-tls
      hosts:
        - "cloud.example.com"

nextcloud:
  host: cloud.example.com
  existingSecret:
    enabled: true
    secretName: nextcloud-admin-credentials
    usernameKey: username
    passwordKey: password
    tokenKey: serverInfoToken

  # update to the latest version of nextcloud
  update: 1

  containerPort: 9000

  securityContext:
    runAsUser: 82
    runAsGroup: 82
    # runAsNonRoot: true

  podSecurityContext:
    runAsUser: 82
    runAsGroup: 82
    fsGroup: 82

  extraInitContainers:
    - name: change-data-dir-permissions
      image: alpine:latest
      command:
        - /bin/sh
        - -c
        - "chmod 770 -R /var/www/html/data && chown 82:www-data -R /var/www/html"
      volumeMounts:
        - mountPath: /var/www/
          name: nextcloud-main
          subPath: root
        - mountPath: /var/www/html
          name: nextcloud-main
          subPath: html
        - mountPath: /var/www/html/data
          name: nextcloud-main
          subPath: data
        - mountPath: /var/www/html/config
          name: nextcloud-main
          subPath: config
      securityContext:
        runAsUser: 0
        runAsGroup: 0
        runAsNonRoot: false
        seccompProfile:
          type: RuntimeDefault

  extraEnv:
    # s3 parameters
    - name: "OBJECTSTORE_S3_HOST"
      valueFrom:
        secretKeyRef:
          name: nextcloud-s3-credentials
          key: S3_HOSTNAME

    - name: "OBJECTSTORE_S3_BUCKET"
      valueFrom:
        secretKeyRef:
          name: nextcloud-s3-credentials
          key: S3_BUCKET

    - name: "OBJECTSTORE_S3_KEY"
      valueFrom:
        secretKeyRef:
          name: nextcloud-s3-credentials
          key: S3_USER

    - name: "OBJECTSTORE_S3_SECRET"
      valueFrom:
        secretKeyRef:
          name: nextcloud-s3-credentials
          key: S3_PASSWORD

    - name: "OBJECTSTORE_S3_PORT"
      value: "443"

    - name: "OBJECTSTORE_S3_SSL"
      value: "true"

    - name: "OBJECTSTORE_S3_REGION"
      value: eu-west-1

    - name: "OBJECTSTORE_S3_USEPATH_STYLE"
      value: "true"

    - name: "PGSSLCERT"
      value: /etc/secrets/nextcloud/tls.crt

    - name: "PGSSLKEY"
      value: /etc/secrets/nextcloud/tls.key

    - name: "PGSSLROOTCERT"
      value: /etc/secrets/ca/ca.crt

  extraVolumes:
    - name: postgres-ca
      secret:
        secretName: nextcloud-postgres-server-ca-key-pair
        defaultMode: 0640

    - name: postgres-client-certs
      secret:
        secretName: nextcloud-postgres-nextcloud-cert
        defaultMode: 0640

  extraVolumeMounts:
    - name: postgres-ca
      mountPath: /etc/secrets/ca

    - name: postgres-client-certs
      mountPath: /etc/secrets/nextcloud

  phpConfigs:
    www.conf: |-
      [www]
      security.limit_extensions = .php .css .js .html
      listen = 127.0.0.1:9000
      pm = dynamic
      pm.max_children = 350
      pm.start_servers = 80
      pm.min_spare_servers = 80
      pm.max_spare_servers = 268

  configs:
    logging.config.php: |-
      <?php
      $CONFIG = array (
        'log_type' => 'file',
        'logfile' => 'nextcloud.log',
        'loglevel' => 1,
        'logdateformat' => 'F d, Y H:i:s'
        );

    s3.config.php: |-
      <?php
      $CONFIG = array (
        'objectstore' => [
          'class' => '\\OC\\Files\\ObjectStore\\S3',
          'arguments' => [
                'bucket' => getenv('OBJECTSTORE_S3_BUCKET'),
                'hostname' => getenv('OBJECTSTORE_S3_HOST'),
                'key' => getenv('OBJECTSTORE_S3_KEY'),
                'secret' => getenv('OBJECTSTORE_S3_SECRET'),
                'region' => getenv('OBJECTSTORE_S3_REGION'),
                'port' => getenv('OBJECTSTORE_S3_PORT'),
                'use_ssl' => true,
                'use_path_style' => true,
                'autocreate' => false,
                'verify_bucket_exists' => false,
          ]
        ]
      );

    proxy.config.php: |-
      <?php
      $CONFIG = array (
        'trusted_proxies' => array(
          0 => '127.0.0.1',
          1 => '10.0.0.0/8'
        ),
        'forwarded_for_headers' => array('HTTP_X_FORWARDED_FOR'),
      );

nginx:
  enabled: false

internalDatabase:
  enabled: false

externalDatabase:
  enabled: true
  type: postgresql
  host: "nextcloud-postgres-rw.nextcloud.svc:5432;sslmode=verify-full;sslrootcert=/etc/secrets/ca/ca.crt;sslcert=/etc/secrets/nextcloud/tls.crt;sslkey=/etc/secrets/nextcloud/tls.key"
  database: nextcloud
  existingSecret:
    enabled: true
    secretName: nextcloud-pgsql-credentials
    usernameKey: username
    passwordKey: password

postgresql:
  enabled: false

redis:
  enabled: false

cronjob:
  enabled: false

service:
  type: ClusterIP
  port: 9000
  nodePort: nil

persistence:
  enabled: true
  existingClaim: nextcloud-config
  nextcloudData:
    enabled: false

livenessProbe:
  enabled: false
readinessProbe:
  enabled: false
startupProbe:
  enabled: false
hpa:
  enabled: false

metrics:
  enabled: false

rbac:
  enabled: true

This results in Nextcloud coming up and initializing, and the ingress is up and healthy, however, when I try to go to nextcloud in the browser for cloud.example.com nextcloud throws a 404 for index.php in the pod logs and 404 for /login in the pod logs. The nextcloud app itself is healthy as it's created it's database and I'm able to use the occ command to download and install apps too, but the actual web interface is still not available and I'm not sure why 🤔

In the browser I see:

File not found. 

In the nextcloud pod logs I see:

Initializing nextcloud 27.1.4.1 ...

New nextcloud instance

Installing with PostgreSQL database

=> Searching for scripts (*.sh) to run, located in the folder: /docker-entrypoint-hooks.d/pre-installation

Starting nextcloud installation

Nextcloud was successfully installed

Setting trusted domains…

System config value trusted_domains => 1 set to string cloud.buildstars.online

=> Searching for scripts (*.sh) to run, located in the folder: /docker-entrypoint-hooks.d/post-installation

Initializing finished

=> Searching for scripts (*.sh) to run, located in the folder: /docker-entrypoint-hooks.d/before-starting

[01-Dec-2023 09:59:50] NOTICE: fpm is running, pid 1

[01-Dec-2023 09:59:50] NOTICE: ready to handle connections

10.42.0.7 -  01/Dec/2023:10:06:19 +0000 "GET /index.php" 404

10.42.0.7 -  01/Dec/2023:10:06:25 +0000 "GET /index.php" 404

10.42.0.7 -  01/Dec/2023:10:06:32 +0000 "GET /login" 404

That's all the time I have for checking into this right now, but curious if anyone else makes it any further. good luck!

[Routhinator]

To make this work, you need to define all of the "location" elements from the nginx config provided by Nextcloud in the ingress definition https://github.com/nextcloud/helm/blob/main/charts/nextcloud/templates/nginx-config.yaml

In an ingress the equivalent for each of those statements is a path, and currently in this chart those are not overridable in a way that allows the addition of additonal paths: https://github.com/nextcloud/helm/blob/main/charts/nextcloud/templates/ingress.yaml

In addition to this, this block https://github.com/nextcloud/helm/blob/main/charts/nextcloud/templates/nginx-config.yaml#L118 and the fonts one below it a problematic. FPM is a php execution server and not a file server, much like it's pythonic "WSGI" server counterparts - they are not for serving static files like images or fonts. This is always handled by the web proxy, whether it's PHP in Apache, or FPM behind an HTTP proxy.

With the included nginx, the static files in the html directory are available to it because the volume mounts containing the static files are mounted in a location such that the path can be tried first by NGINX before forwarding the request to FPM
https://github.com/nextcloud/helm/blob/main/charts/nextcloud/templates/deployment.yaml#L202

I've managed Kube a while but honestly have never looked into mounts being presented to an ingress controller for such a purpose - My search based on the ideas in this issue lead me to the official response, which is what I expected: nginxinc/kubernetes-ingress#323 (comment)

The Ingress Controller is not designed to serve static content. If you'd like to serve static content, consider having a separate Service for it.

See nginxinc/kubernetes-ingress#257 (comment)

And from the original comment in 2018:

the Ingress controller is not designed to serve static content. If you'd like to serve static content, consider having a separate Service for it

Honestly, this feels like the right response. Allowing an ingress controller to mount volumes would require it to have access to mount volume resources in all namespaces, versus today where it just has rights to mount TLS secrets and route to services in those namespaces.

This issue is a dead end in my eyes and that's a good thing. If you are interested in not having NGINX behind NGINX, you could look at alternative ingress controllers that are not built on webservers and are purely proxies - for example the haproxy ingress controller, or better yet - Cilium.

========

As an additional note, if Nextcloud (or any service you have in PHP-FPM) does not need to serve static files, then you can certainly just use ingress-nginx to proxy it.

If you are adamant about wanting to reduce service-specific web proxies for static files in your cluster, then going with a non-http ingress controller is a first step, then you can deploy your own proxy-pod and configure it to mount all required volumes containing static files and understand all services it needs to proxy. Effectively, build a proxy tier for all your applications that need static file serving and have any ingress for apps that have static files route to it instead of the application.

I feel like maybe that type of web proxy service may have a solution in K8S land that is not part of an Ingress service, but I'm not aware of one. I tend to prefer idempotent separation of concerns and would not want to have to roll proxies that front many services in order to mount another volume. You're also going to have to use a RWM volume type for that which presents other challenges.

[provokateurin]

@Routhinator thanks a lot for your insights! I agree with you, the ingress controller should not mount the volumes and have that level of acces. In my eyes we can close this issue then

[jessebot]

Agree with @Routhinator and @provokateurin so closing this :) If someone has any further insights to provide, feel free to open a GitHub Discussion or continue commenting on this issue. If things change in the future with regards to the new gateway api or something, we can talk about reopening this.