Allow template defaults in case of failed generator

[tjkirch]

Fixes #404.

If we can't determine max-pods because we fail to talk to IMDS or fail to read/parse the max-pods file, we want to have a fallback (the Kubernetes default) rather than leaving Kubernetes entirely broken.

This includes a few related commits that allow us to do it cleanly, and to handle other similar cases that come up.

• Allow setting-generator programs to exit 2 to mean "leave the setting empty" so we can use a template default, and give them the option to try again later. (As always, if the user specified the setting, sundog won't try to override it.)
  • ...and do this for max-pods.
• Add a default template helper that allows easier specification of defaults in templates, rather than big conditionals with repetition of the setting.
• Add a default value for max-pods that matches Kubernetes upstream - 110.

Testing done:

Determining max-pods via IMDS still works:

bash-5.0# tail -n1 /etc/kubernetes/kubelet/config
MaxPods: 29

If I remove the setting and remove the max-pods file...

bash-5.0# rm /var/lib/thar/datastore/current/live/settings/kubernetes/max-pods
bash-5.0# mount -o size=1M -t tmpfs tmpfs /usr/share/eks

...then we use the template default; we can see thar-be-settings using it:

bash-5.0# thar-be-settings -v -v
...
Oct 16 19:39:26.149 DEBUG thar_be_settings::config: Rendering kubelet-config
Oct 16 19:39:26.149 TRACE thar_be_settings::helpers: Starting default helper
Oct 16 19:39:26.149 TRACE thar_be_settings::helpers: Template name: kubelet-config
Oct 16 19:39:26.149 TRACE thar_be_settings::helpers: Number of params: 2
Oct 16 19:39:26.149 TRACE thar_be_settings::helpers: Default value if key is not set: 110
Oct 16 19:39:26.149 DEBUG thar_be_settings::config: Rendering kubelet-kubeconfig
Oct 16 19:39:26.149 TRACE thar_be_settings::config: Rendered configs: [RenderedConfigFile { path: "/etc/kubernetes/kubelet/env", rendered: "...SNIP... MaxPods: 110\n" }, RenderedConfigFile ...]

The default shows up:

bash-5.0# tail -n1 /etc/kubernetes/kubelet/config 
MaxPods: 110

sundog/pluto work again if we restore things:

bash-5.0# umount /usr/share/eks/
bash-5.0# rm /var/lib/thar/datastore/current/pending/settings/kubernetes/max-pods
bash-5.0# sundog -v -v
...
Oct 16 19:58:26.495 TRACE sundog: Settings to PATCH to /settings: {"kubernetes":{"max-pods":"29"}}

[zmrow]

⚡️

[tjkirch]

Rebased on top of develop to fix conflict with #419.