k8s-home

This repository contains source code for my own homelab's kubernetes instance. It is used to define & configure deployed apps, build charts that are then used by gitops mechanisms to deploy them on my cluster.

Thanks to cdk8s, the project builds helm charts & apps charts using golang code and stores them in the generated branch. This branch is then used by Flux CD to reconcile the k8s state with the built charts.

Setup

The kubernetes cluster is installed on multiple Mini PCs running Fedora, using k3s.

Installed apps

Most important apps installed on my cluster are:

• longhorn is used to provide distributed physical volumes across the cluster;
• vault & External Secrets Operator are used for most secrets. A few secrets are deployed using sealed-secrets, to unlock encrypted volumes;
• dex-idp is linked with my personal gitea instance to provide oauth based SSO to services supporting it;
• There also is an authentik instance linked with a bunch of services for SSO authentication and authorization;
• traefik-forward-auth protects a few services behind an oauth authn/authz firewall for apps not linked to dex-idp;
• cert-manager generates TLS certificates for ingresses on demand;
• PostgreSQL, Minio, ScyllaDB, MariaDB operators are installed to provide databases instances;
• NATS handles message queues;
• kube-prometheus-stack, along with grafana, provides metrics monitoring;
• blackbox-exporter is installed too;
• Grafana's loki with promtail are setup to aggregate and store logs;
• tempo for traces processing;
• linkerd, Kyverno, Tekton, trivy and temporal are present in the charts for testing;
• A bunch of end-users apps are installed, such as:
  • wallabag, a visual to-read list;
  • freshrss, a RSS aggregator;
  • privatebin, a secure pastebin;
  • paperless-ngx provides document management;
  • yopass, to share secrets securely;
  • bookstack, a platform to organise and share information;
  • IT-Tools, some handy tools for engineers;
  • vaultwarden, a bitwarden compatible password manager;
  • send, simple, private file sharing;
  • snippetbox, a code snippet portal;
  • excalidraw, virtual collaborative whiteboard;
  • wikijs, wiki as stated in its name;
  • redmine, because project management;
  • microbin another pastebin kind of instance;
  • memos for note taking;
  • opengist as a github gist clone.

Deployment

When a PR is merged in main, a Drone CI-based pipeline is started to build cdk8s dependencies, then build charts, and finally push generated charts in the generated branch. This branch is pulled every 2 minutes by flux. See .drone.yml.

Maintenance notes

Initial installation with flux

Initial installation of Flux CD comes with the flux binary. The whole installation procedure is available on the official website.

Please start fluxcd:

flux bootstrap git \
  --url=ssh://[email protected]/mycroft/k8s-home.git \
  --private-key-file=/tmp/rsa \
  --branch=generated \
  --path=generated/

Upgrading flux

Dump key file:

k get secret -n flux-system -o yaml flux-system | yq .data.identity -r | base64 -d > /tmp/rsa

Then re-run the bootstrap command as seen in the Installation section.

Upgrade services

To check outdated Helm charts & images, use:

go build && ./k8s-home -check-version

Services must be updated by modifying versions in source code. Note all containers are not checked in with this command.

Security

Secrets can be either stored in source code thanks to sealed-secrets or in vault and deployed as a Secret in Kubernetes with external-secrets.

Ingresses can be protected thanks to traefik-forward-auth which perform an Oauth process using my own private and external gitea instance.

Services

Traefik tweaks

traefik should/must be reconfigured to redirect all http:// requests to https://.

On master node, edit /var/lib/rancher/k3s/server/manifests/traefik-config.yaml and add the following snippet in valuesContent section:

  valuesContent: |-
    ports:
      web:
        redirectTo: websecure

k3s will automatically redeploy the helm chart.

Also, it should allow IngressRoutes to use middlewares from another namespaces the ingressroutes is in (disabled by default)

    providers:
      kubernetesCRD:
        allowCrossNamespace: true

Note: Do not edit traefik.yaml. It is overwritten on restart.

Viewing traefik dashboard

port-forward the 9000 port and reach http://localhost:9000/dashboard/:

> k port-forward -n kube-system (k get pods -n kube-system | grep ^traefik | cut -d' ' -f1) 9000:9000
Forwarding from 127.0.0.1:9000 -> 9000
Forwarding from [::1]:9000 -> 9000

Kubernetes-Dashboard

Generate a login and use it on https://kubernetes-dashboard.services.mkz.me/

kubectl -n kubernetes-dashboard create token admin

You need a recent kubectl binary to be able to do this!

This is fully described on https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md

See https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/README.md#login-view

Vault & External-Secrets

Vault was manually configured to allow kubernetes based authentication and linking a custom policy to allow External-Secrets queries.

As we're using transient service account token, there is no longer need to have a service account secret and to maintain it. More about this part can be found in the external-secrets docs about hashicorp vault provider.

Documentation about this can be found at https://developer.hashicorp.com/vault/docs/auth/kubernetes

Most likely what to be done is:

$ vault auth enable kubernetes

$ vault write auth/kubernetes/config \
    kubernetes_host=https://10.43.0.1:443

$ vault write external-secrets readwrite /path/to/external-secrets.vault.hcl

$ vault write auth/kubernetes/role/external-secrets \
    bound_service_account_names=external-secrets \
    bound_service_account_namespaces=external-secrets \
    policies=external-secrets \
    ttl=24h

This should be sufficient to create valid secretstores & externalsecrets objects.

Notes

Force a HelmRelease reconcile after an undetected change

Add/edit the annotation:

kubectl annotate --overwrite -n monitoring helmrelease/prometheus reconcile.fluxcd.io/requestedAt="$(date +%s)"

On HelmRelease 'install retries exhausted'

After fixing the issue, suspend & resume the installation:

> flux suspend hr -n kubernetes-dashboard kubernetes-dashboard
► suspending helmrelease kubernetes-dashboard in kubernetes-dashboard namespace
✔ helmrelease suspended

> flux resume hr -n kubernetes-dashboard kubernetes-dashboard
► resuming helmrelease kubernetes-dashboard in kubernetes-dashboard namespace
✔ helmrelease resumed
◎ waiting for HelmRelease reconciliation
✔ HelmRelease reconciliation completed
✔ applied revision 5.11.0

Install whatismyip

It is required to not use the default externalTrafficPolicy to be able to get real-ip.

Please use the following configuration:

# cat /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--serverstransport.insecureskipverify=true"
    service:
      spec:
        externalTrafficPolicy: Local

# kubectl apply -f /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
# kubectl get svc -o yaml -n kube-system traefik | grep -i policy
  externalTrafficPolicy: Local
  internalTrafficPolicy: Cluster

See https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml for more values.

Creating a secret using Sealed-Secrets

See https://github.com/bitnami-labs/sealed-secrets:

# Create a json/yaml-encoded Secret somehow:
# (note use of `--dry-run` - this is just a local file!)
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o json >mysecret.json

# Warning! Do not forget the namespace & the secret name. By default SealedSecret is very strict about those.

# This is the important bit:
# (note default format is json!)
kubeseal <mysecret.json >mysealedsecret.json

# At this point mysealedsecret.json is safe to upload to Github,
# post on Twitter, etc.

# Eventually:
kubectl create -f mysealedsecret.json

Creating PostgreSQL storage

To create and be able to use a PostgreSQL database, a few steps are required:

• In postgresql.go, add a record in the list. Commit and push the file; It will spawn the database and create secrets in the postgres namespace;
• When database is ready, connect to database with psql to allow user to create tables:

$ k exec -ti -n postgres postgres-instance-0 -- psql -U dex-admin dex
dex=# GRANT CREATE ON SCHEMA public TO PUBLIC;
GRANT

• Retrieve username & password from Secret in the postgres namespace:

$ k get secret -n postgres dex-admin.postgres-instance.credentials.postgresql.acid.zalan.do -o yaml | yq -r .data.username | base64 -d
...
$ k get secret -n postgres dex-admin.postgres-instance.credentials.postgresql.acid.zalan.do -o yaml | yq -r .data.password | base64 -d
...

• Create a record in vault (ie: namespaces/<namespace>/postgresql) and create an username and a password record with the values retrieved in the Secret.

• In the target namespace, create an ExternalSecretStore and an ExternalSecret. There is a lot of examples in the repository.