k8s clusters backed by Flux v2

Kubernetes clusters using the GitOps tool Flux.
The Git repository is the driving the state of the Kubernetes clusters.
The awesome Flux SOPS integration is used to encrypt secrets with age.

📂 Repository structure

The Git repository contains the following directories:

📁
├─📁 apps            # apps available for intallation
├─📁 cluster-apps    # kustomization and overlays for app installations per cluster
│  ├─📁 staging
│  └─📁 production
├─📁 charts          # helm chart repos
├─📁 configs         # configs per cluster
└─📁 base
   ├─📁 flux-system  # flux & gitops operator
   ├─📁 staging      # flux configuration per cluster
   └─📁 production   # flux configuration per cluster

💻  Software

The following apps are installed on the clusters.

Software	Purpose
Flux2	GitOps Tool managing the cluster
Longhorn	Persistent Block Storage Provisioner
NGINX Ingress Controller	Cluster Ingress controller
MetalLB	Bare metal LoadBalancer
Cert-Manager	Letsencrypt certificates with Cloudflare DNS
ExternalDNS	Configure Cloudflare DNS Servers
kube-vip	Virtual IP Load-Balancer for Control Plane High Availability
Kube-Prometheus Stack	Prometheus & Exporters to monitor the cluster
Grafana	Monitoring & Logging Dashboard
Alertmanager	Monitoring Alerts
Grafana Loki	Log aggregation system
System Upgrade Controller	Automated k3s upgrades
Weave GitOps	Powerful WebUI extension to Flux for deployment insights
MinIO	Amazon S3 compatible high Performance Object Storage
Authelia	SSO & 2FA authentication server for Cluster Web Apps
Paperless-ngx	Document management system
Tandoor-Recipes	Recipe Manager and Meal Planner
File Browser	Web File Browser
SFTPGo	Full-featured SFTP Server
Immich	Photo and video management solution
Memos	A privacy-first, lightweight note-taking service
Syncthing	Continuous file synchronization service
Radicale	CalDAV and CardDAV Server
Rancher	Kubernetes Management Dashboard
Homer	Static dashboard for the cluster applications
Bind	Full-featured DNS System
Blocky	Fast and lightweight DNS proxy as ad-blocker
Pod-Gateway	Route traffic through a VPN gateway
Descheduler	Evicts pods to optimize scheduling
Goldilocks	Utility to help identifying good resource requests and limits
X.509 Certificate Exporter	A Prometheus exporter to monitor x509 certificates
SMB CSI Driver	CSI Driver for SMB
kured	Kubernetes Reboot Daemon (only used for Monitoring)
Kubernetes Metrics Server	Source of container resource metrics for Kubernetes

🤖  Automation

Renovate Bot makes sure the components are never outdated.

It creates PullRequests when Helm charts or Docker images have newer versions available and even keeps Flux and k3s up-to-date.

🤝  Thanks

Big shout out to k8s@home and everyone from awesome-home-kubernetes for the inspiration ❤️

📖  Notes

📍 Installation Notes

tl;dr

kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f -
sops -d ./base/flux-system/init/flux-sops-age-secret.sops.yaml | kubectl apply -f -
sops -d ./base/flux-system/init/flux-secret.sops.yaml | kubectl apply -f -
kubectl apply --kustomize=./base/flux-system
kubectl apply --kustomize=./base/staging

1. Pre-create the flux-system namespace

kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f -

4. Add the Flux age key in-order for Flux to decrypt SOPS secrets

sops -d ./base/flux-system/init/flux-sops-age-secret.sops.yaml | kubectl apply -f -

5. (Optional) Add the Flux SSH key in-order for Flux to pull private git repositories

sops -d ./base/flux-system/init/flux-secret.sops.yaml | kubectl apply -f -

5. Install Flux

kubectl apply --kustomize=./base/flux-system

6. Configure Flux

kubectl apply --kustomize=./base/staging

🚧 PostgreSQL Major Version Upgrade

I now use a tianon/postgres-upgrade init-container for PostgreSQL Major Upgrades.

Always take backups, dataloss is possible. Old data gets removed and replaced by output of pg_upgrade.

## Init Container for Major PostgreSQL Upgrades, not needed permanently
initContainers:
- name: pg-upgrade
  image: tianon/postgres-upgrade:15-to-16
  securityContext:
    runAsUser: 0
    runAsGroup: 0
  volumeMounts:
  - name: data
    mountPath: /bitnami/postgresql
  env:
  - name: "PG_OLD"
    value: "15"
  - name: "PG_NEW"
    value: "16"
  command:
  - /bin/bash
  - -c
  - |
    if [[ $(< /bitnami/postgresql/data/PG_VERSION) -eq $PG_NEW ]]; then echo "PostgreSQL is already up2date"; exit 0; fi
    if [[ "$PGBINOLD" != "/usr/lib/postgresql/$PG_OLD/bin" ]]; then echo "Wrong postgres-upgrade image"; exit -1; fi
    if [[ "$PGBINNEW" != "/usr/lib/postgresql/$PG_NEW/bin" ]]; then echo "Wrong postgres-upgrade image"; exit -1; fi
    echo "Upgrading PostgreSQL from $PG_OLD to $PG_NEW"
    cp -r /bitnami/postgresql/data /var/lib/postgresql/$PG_OLD
    usermod -u 1001 postgres
    groupmod -g 1001 postgres
    chown -R postgres:postgres /var/lib/postgresql
    su postgres -c 'PGDATA="$PGDATANEW" eval "initdb $POSTGRES_INITDB_ARGS"'
    cp -p $PG_NEW/data/postgresql.conf $PG_NEW/data/pg_hba.conf $PG_OLD/data/
    chmod 700 $PG_OLD/data
    gosu postgres pg_upgrade
    gosu postgres pg_ctl -D /var/lib/postgresql/$PG_NEW/data -l logfile start
    gosu postgres /usr/lib/postgresql/$PG_NEW/bin/vacuumdb --all --analyze-in-stages
    gosu postgres pg_ctl -D /var/lib/postgresql/$PG_NEW/data -l logfile stop
    rm /var/lib/postgresql/$PG_NEW/data/pg_hba.conf /var/lib/postgresql/$PG_NEW/data/postgresql.conf
    rm -rf /bitnami/postgresql/data
    mv /var/lib/postgresql/$PG_NEW/data /bitnami/postgresql/

🚧 Previous Manual Upgrade Approach

Based on bitnami/charts#1798 (comment)

1. Preparation

export NAMESPACE=selfhosted
export APPLICATION_DEPLOYMENT=paperless-ngx
export POSTGRES_DEPLOYMENT=${APPLICATION_DEPLOYMENT}-postgresql
export POSTGRES_MAJOR_VERSION=12
export POSTGRES_PVC_SIZE=4Gi
export POSTGRES_DB=${APPLICATION_DEPLOYMENT}
export POSTGRES_USERNAME=${APPLICATION_DEPLOYMENT}
export POSTGRES_PASSWORD=yourSecretPassword

Note: POSTGRES_MAJOR_VERSION is the helm version, not postgresql version.

2. Scale down application that uses the database

kubectl scale deployment ${APPLICATION_DEPLOYMENT} -n ${NAMESPACE} --replicas 0

3. Deploy new major version of the database

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

helm install ${POSTGRES_DEPLOYMENT}-upgrade bitnami/postgresql --version ${POSTGRES_MAJOR_VERSION} -n ${NAMESPACE} --wait \
    --set auth.username=${POSTGRES_USERNAME} \
    --set auth.password=${POSTGRES_PASSWORD} \
    --set auth.database=${POSTGRES_DB} \
    --set primary.persistence.size=${POSTGRES_PVC_SIZE}

4. Migrate data to new postgresql deployment

kubectl exec -it ${POSTGRES_DEPLOYMENT}-upgrade-0 -n ${NAMESPACE} -- bash -c "export PGPASSWORD=${POSTGRES_PASSWORD}; time pg_dump -h ${POSTGRES_DEPLOYMENT} -U ${POSTGRES_USERNAME} | psql -U ${POSTGRES_USERNAME}"

From here on you have multiple possibilities, e.g. just use your app with the new db deployment.

I personally prefer to backup the volume of the new DB, uninstall both database deployments & delete their PVCs.

After that I restore the backup of the PVC with the name of the old database deployment & upgrade my Helmrelease version.

helm uninstall ${POSTGRES_DEPLOYMENT}-upgrade -n ${NAMESPACE}
kubectl scale sts ${POSTGRES_DEPLOYMENT} -n ${NAMESPACE} --replicas 0

The volume deletion and restore is done in Longhorn UI. Afterwards helm upgrade for the postgresql deployments can be done.

5. Scale up application that uses the database

kubectl scale deployment ${APPLICATION_DEPLOYMENT} -n ${NAMESPACE} --replicas 1