Skip to content

Machine and k8s node Setup

Jump to bottom
rgaudin edited this page Oct 3, 2024 · 4 revisions

Machine / VM setup

  • Setup a Debian host on a single partition
  • comment-out swap entries in /etc/fstab
  • mount -a
  • timedatectl set-timezone UTC
  • apt update && apt upgrade -y
  • apt install -y vim screen rsync htop ncdu fail2ban tree
  • Edit /etc/motd with hostname ASCII
  • Enhance shell
cat <<EOF | tee -a /etc/profile.d/aliases.sh
export LS_OPTIONS='--color=auto'
eval "\$(dircolors)"
alias ls='ls $LS_OPTIONS'
alias ll='ls $LS_OPTIONS -l'
alias l='ls $LS_OPTIONS -lA'

#Some more alias to avoid making mistakes:
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
EOF
  • Add node's IP and DNS name to bastion's /etc/hosts (otherwise ProxyJump might fail when using DNS name to reach target)
  • Setup IP blocklist (from our bastion and WMF's)
cat <<EOF | tee -a /etc/hosts.allow
sshd: localhost
sshd: bastion.kiwix.org
sshd: 51.159.132.199
sshd: [2001:bc8:1200:570a::1]
sshd: 185.15.56.0/24
EOF
echo "sshd: ALL" | tee -a /etc/hosts.deny
  • if external block storage
lsblk -f
fdisk /dev/xxx # create a single primary Linux partition
mkfs.ext4 -L data /dev/sdb1
mkdir -p /data
echo "UUID=xxxxxxxxx /data ext4 defaults 0 0" | tee -a /etc/fstab
mount -a
ls -l /data

Configure k8s

  • execute Scaleway installation tool

(below is a copy of Scaleway procedure, program is also backed-up at https://dev.kiwix.org/node-agent_linux_amd64_2023_08_31)

Note that this is the new procedure (as of September 2023), only used on system and storage2 nodes for now. Other nodes (stats, services and storage) have been provisioned with an older tool/procedure.

  1. Retrieve the node-agent program (suffix can be either amd64 or arm64 depending on the node architecture).
wget https://scwcontainermulticloud.s3.fr-par.scw.cloud/node-agent_linux_amd64 && chmod +x node-agent_linux_amd64
  1. Export the required environment variables. Replace the values where needed.
export POOL_ID=4a574aa5-737e-4993-961a-1a8d629ee4ea POOL_REGION=fr-par SCW_SECRET_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  1. Execute the program to attach the node to the Multi-cloud pool.
sudo -E ./node-agent_linux_amd64 -loglevel 0 -no-controller
  • Await addition of node on Scaleway (status=ready != status=creating)
http https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER/nodes "X-Auth-Token: $SCW_SECRET_KEY"
  • check node status on cluster
kubectl get nodes -o wide
  • set role labels on node
kubectl label node NODENAME node-role.kubernetes.io/MYROLE=true
kubectl label node NODENAME k8s.kiwix.org/role=MYROLE
  • enable kubelet service
systemctl enable kubelet.service

Special setup for dedicated machines on Hetzner

(configuration below is the one from storage2 node, to be adapted)

If not already, boot the server in Rescue mode.

Install OS with installimage from Hetzner:

  • enable only SSD disks
  • create only /boot + first LVM partition and logical volume for /
  • everything else does not need to be handled by installimage and will be done manually afterwards

Important items of the configuration

DRIVE1 /dev/nvme0n1
DRIVE2 /dev/nvme1n1
# DRIVE3 /dev/sda
# DRIVE4 /dev/sdb
# DRIVE5 /dev/sdc
# DRIVE6 /dev/sdd

SWRAID 1
SWRAIDLEVEL 1

HOSTNAME storage2

PART /boot ext3      1G
PART lvm   ssds_r1   250G

LV ssds_r1 root /    ext4  150G

This will create two raid1 partitions on nvme disks, one (1G, ext3) for /boot, one (250G) for lvm ssds_r1 VG. And inside ssds_r1 VG it will create a 150G LV root, formatted as ext4 and mounted for /. Rest of nvme disks is left unused by installimage just like HDDs.

Create /dev/nvme0n1p3 and /dev/nvme1n1p3 with fdisk:

  • fdisk /dev/nvme0n1
  • n (new partition)
  • p (primary)
  • 3 (third primary partition)
  • t (change type)
  • 3 (third primary partition)
  • fd (Linux RAID autodetected)
  • w (write changes)

Create /dev/md2 and /dev/md3 RAID

mdadm --create /dev/md2 --level=0 --raid-devices=2 /dev/nvme0n1p3 /dev/nvme1n1p3
mdadm --create /dev/md3 --level=6 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

Configure LVM:

vgcreate ssds_r0 /dev/md2
vgcreate hdds /dev/md3
lvcreate -L 30T -n data hdds

Create /data FS:

mkfs.ext4 /dev/hdds/data

Modify fstab and reload

  • create mount point mkdir /data
  • add /dev/hdds/data /data ext4 defaults 0 0 to /etc/fstab
  • systemctl daemon-reload / mount -a

Final fstab content is something like this:

proc /proc proc defaults 0 0
UUID=f2d49190-2037-4734-b094-aa7160f8bab3 /boot ext3 defaults 0 0
/dev/ssds_r1/root  /  ext4  defaults 0 0
/dev/hdds/data  /data  ext4  defaults 0 0

TODO: configure SSD cache (see e.g. https://forum.level1techs.com/t/lvm-raid-with-ssd-cache-guide/149551)

Clone this wiki locally