Skip to content

Cluster Setup

Jump to bottom
benoit74 edited this page Apr 4, 2024 · 9 revisions

Instructions on how-to/how-was setup the k8s Kosmos Cluster at Scaleway, using webUI

  • Create Cluster without pool (those are scaleway)
  • Enable Ingress controller (nginx) via UI or API
  • Download kubeconfig and store as ~/.kube/scw-admin_kiwix-prod.config
  • Add Pool (foreign)
  • Add a node. It will be your first node (it's special but not CP).
  • set CNAME scw.k8s to 58cd205e-743d-4cf2-bf23-db3c7297f675.nodes.k8s.fr-par.scw.cloud.
  • set CNAME api.scw.k8s to 58cd205e-743d-4cf2-bf23-db3c7297f675.api.k8s.fr-par.scw.cloud.
  • why?
    • scw.k8s has one entry per node so requests are are redirected to any of the nodes
    • api.scw.k8s is control plane. should not change but is referenced in config files, so easier maintenance with clear name.
    • k8s CNAME to scw.k8s. That's where we point all domains.
    • allows us to setup a parallel xyz.k8s node or even a different CP (api.xyz.k8s) and test it independently then instantly switch everything by pointing k8s to xyz.k8s
  • Add api.scw.k8s to ControlPlane SAN
  • you'll need some environment variables:
# get those following https://developers.scaleway.com/en/quickstart/
export SCW_ACCESS_KEY=xxx
export SCW_SECRET_KEY=yyy
export KIWIX_PROD_CLUSTER=c7704f7d-c6e9-49b3-9aaf-a04aa4d5a727
export KIWIX_PROD_FOREIGN_POOL=4a574aa5-737e-4993-961a-1a8d629ee4ea
curl -X PATCH -d '{"apiserver_cert_sans": ["api.scw.k8s.kiwix.org"]}' -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$CLUSTER_ID

Our Control Plane is now at https://api.scw.k8s.kiwix.org:6443

  • Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.2/cert-manager.yaml
# verify its working
kubectl get pods --namespace cert-manager

  • Follow rest of HTTP Steup steps (ClusterIssuer)

  • Create superuser

./create-superuser reg

Upgrading Kubernetes

Upgrade control plane

  • Find out version to upgrade to. ⚠️ downgrade impossible.
http https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER/available-versions "X-Auth-Token: $SCW_SECRET_KEY"
  • Upgrade the cluster and node-pools using UI or API
curl -X POST -d '{"version": "1.23.4", "upgrade_pools": true}' -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER/upgrade
  • Verify version and wait until status is ready (after updating)
http https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER "X-Auth-Token: $SCW_SECRET_KEY"

Upgrade unmanaged nodes

(scaleway-managed pools may not require this)

  • disable schedule on node: kubectl cordon $NODE_NAME
  • [inside node – ssh] ./upgrade-node.sh $NEW_VERSION
  • re-enable schedule: kubectl uncordon $NODE_NAME

Upgrade kube-state-metrics image

kube-state-metrics version has to be aligned with k8s version ; there as some upward and downward compatibility, and it is expected that a recent kube-state-metrics will more nicely cooperate with an old k8s cluster than the other way around.

Find the appropriate version for current k8s API version: https://github.com/kubernetes/kube-state-metrics#compatibility-matrix

Update the version in https://github.com/kiwix/k8s/blob/main/grafana/grafana.values.yaml and upgrade the release with helm upgrade (see detailed instructions in https://github.com/kiwix/k8s/blob/main/grafana/README.md#installation--upgrade)

Upgrade nginx ingress controller

Check if a new release if available in https://github.com/kubernetes/ingress-nginx/#supported-versions-table

Check changelog at https://github.com/kubernetes/ingress-nginx/tree/main/changelog

Retrieve deploy.yaml for Scaleway matching your new version, e.g. https://github.com/kubernetes/ingress-nginx/blob/controller-v1.9.4/deploy/static/provider/scw/deploy.yaml

Backport changes to https://github.com/kiwix/k8s/blob/main/nginx-ingress/deploy.yaml

Deploy with kubectl apply -f nginx-ingress/deploy.yaml

Upgrade cert manager

We are using static manifests. See doc at https://cert-manager.io/docs/installation/upgrade/ Check changelog at https://cert-manager.io/docs/releases/

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/<version>/cert-manager.yaml
Clone this wiki locally