See resources for example kube manifests.
apt-get install unattended-upgrades
Disable apt-periodic unattended-upgrades:
sed -i 's/APT::Periodic::Unattended-Upgrade .*/APT::Periodic::Unattended-Upgrade "0";/' /etc/apt/apt.conf.d/20auto-upgrades
Alternatively, you can completely disable apt-periodic, as the upgrade process will also run apt-get update before the unattended-upgrades:
systemctl stop apt-daily.timer apt-daily-upgrade.timer
systemctl disable apt-daily.timer apt-daily-upgrade.timer
The set of allowed origins for package upgrades can either be configured via the system /etc/apt/apt.conf.d/50unattended-upgrades or using a ConfigMap unattended-upgrades.conf, but the default settings are reasonable.
yum install yum-cron
Ensure that the yum-cron service is stopped and disabled, as this will interfere with pharos-host-upgrades:
systemctl stop yum-cron.service
systemctl disable yum-cron.service
The default yum-cron.conf random_sleep = 360 should also be disabled, either via the default system /etc/yum/yum-cron.conf file, or using a ConfigMap yum-cron.conf. The default yum-cron.conf will only download updates, and requires apply_updates = yes to actually upgrade the host.
When configured to run as a kube DaemonSet pod (using KUBE_* envs), the following kube API integrations can be used:
The host upgrades will only run while holding a lock on the kube daemonset, ensuring that only one host upgrades at a time. This lock is also held during a reboot, and released once the pod restarts.
The lock is implemented as a pharos-host-upgrades.kontena.io/lock annotation on the DaemonSet.
If configured with --reboot --drain, the kube node will be drained before rebooting, marking the node as unschedulable and evicting pods to move them to other nodes for the duration of the reboot.
The node will be uncordoned once the host-upgrades pod is restarted, but only if the pharos-host-upgrades.kontena.io/drain annotation was previously set as a result a drain + reboot triggered by the host-upgrades pod. If the pod is restarted while the node was otherwise drained, it will not be uncordoned.
The kube node .Status.Conditions will be updated based on the result of the host upgrades:
The HostUpgrades condition describes the state of the upgrade command itself.
The condition will be True if the upgrade was run and the node is now up to date, with a message describing any packages upgraded during the last run.
The condition will be False if the host is not up to date. This will happen in the RebootRequired case, where the host requires a reboot to finish applying the upgrades.
In case of the upgrade failing, the condition will be Unknown, with a message describing the error.
The HostUpgradesReboot condition will be True if the host requires a reboot to finish applying upgrades, and False otherwise.
- Kubernetes 1.10
- Kubernetes 1.11
The Go binary can be run natively, outside of Docker or Kube:
sudo $GOPATH/bin/pharos-host-upgrades
This mode only requires systemd DBUS access. Kube locks and configs are not supported.
Example docker run options:
docker run --rm --name host-upgrades -v $PWD/config:/etc/host-upgrades -v /run/host-upgrades:/run/host-upgrades -v /var/run/dbus:/var/run/dbus -v /run/log/journal:/run/log/journal --privileged kontena/pharos-host-upgrades
See the example kube resources:
kubectl apply -f ./resources
The DaemonSet configures the KUBE_* envs, enabling the kubernets integrations.
Usage of pharos-host-upgrades:
-alsologtostderr
log to standard error as well as files
-config-path string
Path to configmap dir (default "/etc/host-upgrades")
-drain
Drain kube node before reboot, uncordon after reboot
-host-mount string
Path to shared mount with host. Must be under /run to reset when rebooting! (default "/run/host-upgrades")
-kube-daemonset string
Name of kube DaemonSet (KUBE_DAEMONSET) (default "host-upgrades")
-kube-namespace string
Name of kube Namespace (KUBE_NAMESPACE) (default "kube-system")
-kube-node string
Name of kube Node (KUBE_NODE) (default "ubuntu-xenial")
-log_backtrace_at value
when logging hits line file:N, emit a stack trace
-log_dir string
If non-empty, write log files in this directory
-logtostderr
log to standard error instead of files
-reboot
Reboot if required
-reboot-timeout duration
Wait for system to shutdown when rebooting (default 5m0s)
-schedule string
Scheduled upgrade (cron syntax)
-stderrthreshold value
logs at or above this threshold go to stderr
-v value
log level for V logs
-vmodule value
comma-separated list of pattern=N settings for file-filtered logging
command terminated with exit code 2
See https://godoc.org/github.com/robfig/cron#hdr-CRON_Expression_Format (no seconds field).
Standard crontab with five fields (Minutes, Hours, Day of month, Month, Day of week).
Examples:
15 5 * * *- every day at 05:151 0 * * SUN- every sunday at 01:00@dailyat midnight
Reboot the host after upgrades, if required.
Drain the kube node before rebooting, and uncordon once restarted.
The kube DaemonSet also supports an optional ConfigMap with configuration files for the host OS package upgrade tools. The ConfigMap should be mounted at --config-path=/etc/host-upgrades, and the --host-mount=/run/host-upgrades path should be bind-mounted from the host.
Refer to the host /etc/apt/apt.conf.d/50unattended-upgrades config file shipped by the unattended-upgrades package.
Note overriding the Unattended-Upgrade::Allowed-Origins list requires the use of a #clear directive, to have the list items in the config override those in the default host config, rather than merging the two lists together. See the sample unattended-upgrades.conf for an example:
// Override system /etc/apt/apt.conf.d/50unattended-upgrades
#clear "Unattended-Upgrade::Allowed-Origins";
// Automatically upgrade packages from these (origin:archive) pairs
Unattended-Upgrade::Allowed-Origins {
...
};
Refer to the host /etc/yum/yum-cron.conf config file shipped by the yum-cron package.
Note that the default random_sleep value will delay upgrades across the entire cluster, and should be disabled. See the sample yum-cron.conf for an example:
[commands]
random_sleep = 0
Using the vagrant machines:
$ vagrant up ubuntu
$ vagrant up centos-7
Note that the centos/7 & debian/stretch64 boxes do not support shared folders... vagrant falls back to rsync, so you must vagrant rsync centos-7 or vagrant rsync debian-9 before building the image after any edits.
$ mkdir .kube
$ sudo cat /etc/kubernetes/admin.conf > .kube/config
/vagrant $ docker build -t kontena/pharos-host-upgrades:dev .
/vagrant $ docker run --rm --name host-upgrades --privileged \
-v $PWD/config:/etc/host-upgrades \
-v /run/host-upgrades:/run/host-upgrades \
-v /var/run/dbus:/var/run/dbus \
-v /run/log/journal:/run/log/journal \
kontena/pharos-host-upgrades:dev