maintenance.md

How to maintain neco-containers

This document describes the procedure for updating each container image.

Besides on-demand container updates, we have two regular renewal operations: Kubernetes Update and Regular Update. The target container of these operations have the following badges, so check before the operations.

In case of components whose Go source code are in neco-containers, all dependent Go modules should be updated if there is no special reason. Kubernetes-related modules such as client-go may be newer than the Kubernetes to be updated. For example, it is acceptable that client-go is v0.30 and Kubernetes is v1.29.

1. Kubernetes Update ()
   • Upgrade of Kubernetes. Besides the related components of Kubernetes, update the containers managed by CKE and some go modules.
2. Regular Update ()
   • Update in every quarter. Keeping up with the upstream version and updating the ubuntu base image.
3. CSA Update ()
   • Update by CSA team.
4. No Need Update ()
   • Used as a PoC, so regular updates are not required.

---

• admission (neco-admission)
• alertmanager
• argocd
• argocd-image-updater
• bird
• blackbox_exporter
• bmc-reverse-proxy
• bpf-map-pressure-exporter
• cadvisor
• cep-checker
• ceph
  • Create a patched image from the specific version
• ceph-extra-exporter
• cephcsi
• cert-manager
• chrony
• cilium
• cilium-certgen
• cilium-operator-generic
• configmap-reload
• contour
• coredns
• csi sidecars
• dex
• envoy
• etcd
• external-dns
• fluent-bit
• golang-all (golang for combinations of versions and platforms)
• gorush
• grafana
• grafana-operator
• haproxy
• heartbeat
• hubble
• hubble-relay
• hubble-ui
• kube-metrics-adapter
• kube-state-metrics
• kube-storage-version-migrator
• kubernetes
• local-pv-provisioner
• loki
• machines-endpoints
• memcached
• memcached_exporter
• meows-dctest-runner
• meows-neco-runner
• moco-switchover-downtime-monitor
• opentelemetry-collector
• pause
• pod-delete-rate-limiter
• pomerium
• prometheus-adapter
• prometheus-config-reloader
• promtail
• promtail-debug
• pushgateway
• redis
• registry
• rook
• s3gw
• sealed-secrets
• serf
• squid
• squid-exporter
• stakater/Reloader
• tcp-keepalive
• teleport-node
• tempo
• testhttpd
• trust-manager
• trust-packages
• ttypdb
• unbound
• unbound_exporter
• vault
• victoriametrics
• victoriametrics-operator

---

admission (neco-admission)

In Kubernetes update:

1. Update the following version variables in Makefile.

   • CONTROLLER_TOOLS_VERSION
   • KUSTOMIZE_VERSION
   • ENVTEST_K8S_VERSION

2. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.

3. Generate code and manifests.

   cd $GOPATH/src/github.com/cybozu/neco-containers/admission
   make generate manifests
   # Commit, if there are any updated files.

4. Confirm build and test are green.

   make build test

5. Update TAG file.

In Regular update, do the following as part of the update of each CRD-providing product:

1. Update a matching version variable from the following in Makefile.

   • CONTOUR_VERSION
   • ARGOCD_VERSION
   • GRAFANA_OPERATOR_VERSION

2. Modify the code to match the new CRDs if CRDs are changed.

   • The code which depended on the CRDs are in the hook directory.
   • And let's use Unstructured instead of use golang library. Take a look at this PR.

3. Generate code and manifests.

   cd $GOPATH/src/github.com/cybozu/neco-containers/admission
   make clean
   make generate manifests
   # Commit, if there are any updated files.

4. Confirm build and test are green.

   make build test

5. Update TAG file.

alertmanager

1. Check the release page.
   • https://github.com/prometheus/alertmanager/releases
2. Check the upstream Dockerfile. If there are any updates, update our Dockefile.
   • https://github.com/prometheus/alertmanager/blob/vX.Y.Z/Dockerfile
3. Update version variables in Dockerfile.
4. Update BRANCH and TAG files.

argocd

1. Check releases for changes.
2. Check hack/tool-versions.sh for the tools versions.
   • https://github.com/argoproj/argo-cd/blob/vX.Y.Z/hack/tool-versions.sh
3. Update tool versions in Dockerfile.
   • Kustomize
   • Helm
4. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/argoproj/argo-cd/blob/vX.Y.Z/Dockerfile
5. Update version variables in Dockerfile.
   • Update ARGOCD_VERSION, KUSTOMIZE_VERSION and HELM_VERSION.
6. Update BRANCH and TAG files.
7. Follow maintenance instructions for neco-admission if needed.

Note

ArgoCD depends on dex,Redis,HAProxy. So browse the following manifests and update dex,redis,haproxy images next.

• https://github.com/argoproj/argo-cd/blob/vX.Y.Z/manifests/base/dex/argocd-dex-server-deployment.yaml
• https://github.com/argoproj/argo-cd/blob/vX.Y.Z/manifests/base/redis/argocd-redis-deployment.yaml
• https://github.com/argoproj/argo-cd/blob/vX.Y.Z/manifests/ha/install.yaml

Note

ArgoCD's Application objects are validated by neco-admission. If Application CRD has been changed, you may need to update neco-admission.

argocd-image-updater

1. Check releases for changes.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/argoproj-labs/argocd-image-updater/blob/vX.Y.Z/Dockerfile
3. Update version variables in Dockerfile.
   • Update ARGOCD_IMAGE_UPDATER_VERSION.
4. Update TAG file.

bird

1. Check the releases page in the official website.
2. Update BIRD_VERSION variable in Dockerfile.
3. Update image tag in README.md.
4. Update BRANCH and TAG files.

blackbox_exporter

1. Check the release page.
2. Update BLACKBOX_EXPORTER_VERSION in Dockerfile.
3. Update BRANCH and TAG files.

bmc-reverse-proxy

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.

2. Confirm test are green.

   make test

3. Update image tag in bmc-reverse-proxy.yaml.

4. Update TAG file.

bpf-map-pressure-exporter

TBD

cadvisor

1. Check the release page.
2. Check the upstream build files. If there are any updates, update our Dockerfile.
   • https://github.com/google/cadvisor/blob/vX.Y.Z/Makefile
   • https://github.com/google/cadvisor/blob/vX.Y.Z/build/release.sh
   • https://github.com/google/cadvisor/blob/vX.Y.Z/build/build.sh
   • https://github.com/google/cadvisor/blob/vX.Y.Z/deploy/Dockerfile
3. Update CADVISOR_VERSION in Dockerfile
4. Update TAG file.

cep-checker

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.
2. Update cilium and cilium-cli version in Makefile and go.mod to the version used by neco.
3. Update TAG by incrementing the patch revision, e.g. 1.0.1, 1.0.2, ...

ceph

1. Check the release page.
2. Check the build ceph document and README.md.
   1. If other instructions are needed for ceph/build.sh, add the instructions.
   2. If there are ceph runtime packages or required tool changes, update Dockerfile.
3. Update the version argument on the build-ceph job in the CircleCI main workflow and the build_ceph job in the Github Actions main workflow.
4. Update BRANCH and TAG files.

Note

The rook image is based on the ceph image. So upgrade the rook image next.

Create a patched image from the specific version

When you want to create a new image with patches to the specific version of Ceph, follow these steps.

1. Create a branch with the name ceph-vX.Y.Z from the commit you want, and push it.
   • You must follow the branch naming convention to activate the image build and push jobs.
   • If the branch already exists, you can skip this step.
2. Create a PR to the branch ceph-vX.Y.Z, and merge it.

ceph-extra-exporter

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.
2. Upgrade base images in Dockerfile.
3. Update the TAG files accordingly.

cephcsi

1. See Rook's values.yaml file of the appropriate tag and check the version of cephcsi.
2. Update CSI_IMAGE_VERSION in Dockerfile with the value which you checked in the previous step.
3. Update BASE_IMAGE in Dockerfile if necessary.
   • If BASE_IMAGE is too old, the build may fail.
   • You should also check BASE_IMAGE in the upstream build.env file of the appropriate tag.
4. See the upstream Dockerfile of the appropriate tag, and update our Dockerfile if necessary.
5. Update BRANCH and TAG files.

Note

Because cephcsi container is build based on the ceph container, build the ceph container first if necessary.

cert-manager

1. Check releases for changes.
2. Check whether manually applied patches have been included in the new release and remove them accordingly.
   1. If patches are still needed, synchronize the forked repository (https://github.com/cybozu-go/cert-manager).
   2. Create and checkout a new branch named vX.Y.Z-neco from the tag named vX.Y.Z.
   3. Cherry-pick the commit included patches and create a new tag named vX.Y.Z-neco-longtimeout.1.
   4. Push it.
3. Update CERT_MANAGER_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update BRANCH and TAG files.

chrony

1. Check the release note.
2. Update CHRONY_VERSION in Dockerfile.
3. Update image tag in README.md.
4. Update BRANCH and TAG files.

cilium

1. Check the releases page for changes.
2. If necessary, update cilium-proxy_version and image-tools_version parameters in the .github/workflows/cilium.yaml.
   1. The version for envoy is referenced in the Dockerfile for cilium in the source repository and is a commit hash from cilium/proxy
   2. Check the upstream Dockerfile and update the .github/actions/build_cilium-envoy/action.yaml as needed.
      • Dockerfile.builder that includes installation of dependencies and Bazel.
      • Dockerfile that builds and installs cilium-envoy.
   3. For image-tools_version, use the latest commit hash from cilium/image-tools
   4. Check the upstream Dockerfile and update the .github/actions/build_cilium-image-tools/action.yaml as needed.
      • compilers/Dockerfile that includes installation of dependencies.
      • bpftool/Dockerfile
      • llvm/Dockerfile
      • iproute2/Dockerfile
3. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/cilium/cilium/blob/vX.Y.Z/images/cilium/Dockerfile
4. Check whether manually applied patches have been included in the new release and remove them accordingly.
5. Update the BRANCH and TAG files accordingly.

Note

The cilium-operator-generic and hubble-relay images should be updated at the same time as the cilium image for consistency.

cilium-certgen

1. Check the releases page for changes.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/cilium/certgen/blob/vX.Y.Z/Dockerfile
3. Update the BRANCH and TAG files accordingly.

cilium-operator-generic

1. Check the releases page for changes.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/cilium/cilium/blob/vX.Y.Z/images/operator/Dockerfile
3. Update the BRANCH and TAG files accordingly.

Note

The cilium-operator-generic image should be updated at the same time as the cilium image for consistency.

configmap-reload

1. Check the tags page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/jimmidyson/configmap-reload/blob/vX.Y.Z/Dockerfile
3. Update CONFIGMAP_RELOAD_VERSION in Dockerfile
4. Update BRANCH and TAG files.

contour

Note

Contour uses Envoy as a "data plane." Keep version correspondence between the contour and envoy images. Check the compatibility matrix below.

• Contour Compatibility Matrix

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/projectcontour/contour/blob/vX.Y.Z/Dockerfile
3. Update CONTOUR_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update BRANCH and TAG files.
6. Follow maintenance instructions for neco-admission if needed.

Note

Contour's HTTPProxy objects are validated by neco-admission. If HTTPProxy CRD has been changed, you may need to update neco-admission.

coredns

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/coredns/coredns/blob/vX.Y.Z/Dockerfile
3. Update COREDNS_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update BRANCH and TAG files.

csi sidecars

This section applies to the following containers. These containers are maintained similarly.

• csi-attacher
• csi-node-driver-registrar
• csi-provisioner
• csi-resizer
• csi-snapshotter

1. See Rook's values.yaml file of the appropriate tag and check the version of csi sidecars.
2. Update VERSION in Dockerfile with the value which you checked in the previous step.
3. See the upstream Dockerfile of the appropriate tag, and update our Dockerfile if necessary. The upstream Dockerfile is listed below.
   • csi-attacher
   • csi-node-driver-registrar
   • csi-provisioner
   • csi-resizer
   • csi-snapshotter
4. update image tag in Dockerfile if necessary.
5. Update BRANCH and TAG files.

Note

You can choose the latest stable Ubuntu image for runtime. upstream uses distroless as the base image for runtime, while Neco uses Ubuntu for easier debugging.

Note

You may choose the latest docker image for the build, regardless of the upstream go version. The current go compiler builds with the language version and toolchain version based on the go version specified in the go.mod file. There is no need to use an older version of the image to match go.mod. As a known issue, the upstream build script warns that test-gofmt and test-vendor are known to be sensitive to the version of Go.. However, we use the latest docker image unless the test fails.

dex

Note

This image is used by ArgoCD. So browse the following manifest and check the required version. If the manifest uses version a.b.c, we should use version a.b.d where d >= c. Don't use a newer minor version.

• https://github.com/argoproj/argo-cd/blob/vX.Y.Z/manifests/base/dex/argocd-dex-server-deployment.yaml

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/dexidp/dex/blob/vX.Y.Z/Dockerfile
3. Update DEX_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update BRANCH and TAG files.

envoy

Note

Envoy is managed by Contour so update to the supported version. See the below.

• Contour Compatibility Matrix

1. Check the release page.
2. Update clang_archive_path in .github/workflows/main.yaml if you want to update the clang version.
3. Update image tag in README.md.
4. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.
5. Update BRANCH and TAG files.

etcd

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/etcd-io/etcd/blob/vX.Y.Z/Dockerfile-release.amd64
3. Update ETCD_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update BRANCH and TAG files.

external-dns

1. Check the release page.
2. Check the upstream .ko.yaml. If there are any updates, update our Dockerfile.
   • https://github.com/kubernetes-sigs/external-dns/blob/vX.Y.Z/.ko.yaml
3. Update EXTERNALDNS_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update TAG file.

fluent-bit

1. Check the release page.
2. Update FLUENT_BIT_VERSION in Dockerfile.
3. Update TAG.

golang-all (golang for combinations of versions and platforms)

1. Check the release history.
2. Update GO_VERSION in Dockerfile.
3. Update BRANCH and TAG.

gorush

Ignore!!!

grafana

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/grafana/grafana/blob/vX.Y.Z/Dockerfile
   • Check JS_IMAGE in the Dockerfile
3. Update GRAFANA_VERSION in Dockerfile.
4. Update installation of Node.js in Dockerfile according to JS_IMAGE if necessary.
5. Update TAG.

grafana-operator

1. Check the release page.
2. Check the upstream build procedure (Makefile, Dockerfile, .ko.yaml, etc). At the point of v5.4.1, grafana-operator is built by ko with its default configuration. If there are any updates, update our Dockerfile.
   • https://github.com/grafana/grafana-operator/tree/vX.Y.Z
3. Update VERSION in Dockerfile.
4. Update TAG.
5. Update GRAFANA_OPERATOR_VERSION in admission/Makefile.
6. Follow maintenance instructions for neco-admission if needed.

Note

Grafana Operator's GrafanaDashboard objects are validated by neco-admission. If GrafanaDashboard CRD has been changed, you may need to update neco-admission.

haproxy

Note

This image is used by ArgoCD. So browse the following manifest and check the required version. If the manifest uses version a.b.c, we should use version a.b.d where d >= c. Don't use a newer minor version.

• https://github.com/argoproj/argo-cd/blob/vX.Y.Z/manifests/ha/install.yaml

1. Check the release notes in the official site.
   • v2.6.x: https://github.com/docker-library/haproxy/blob/master/2.6/Dockerfile
2. Update HAPROXY_SHA256 in Dockerfile, SHA256 hash in http://www.haproxy.org/download
3. Update BRANCH and TAG files.

heartbeat

Only the base image and module dependency should be updated.

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.
2. Update TAG by incrementing the patch revision, e.g. 1.0.1, 1.0.2, ...

hubble

1. Check the releases page for changes.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/cilium/hubble/blob/vX.Y.Z/Dockerfile
3. Update the BRANCH and TAG files accordingly.

hubble-relay

1. Check the releases page for changes.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/cilium/cilium/blob/vX.Y.Z/images/hubble-relay/Dockerfile
3. Update the BRANCH and TAG files accordingly.

Note

The hubble-relay image should be updated at the same time as the cilium image for consistency.

hubble-ui

1. Check the releases page for changes.
2. Update the BRANCH and TAG files accordingly.
3. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/cilium/cilium/blob/vX.Y.Z/images/hubble-relay/Dockerfile
4. hubble-ui depends on nginx. As such, it may be also be necessary to bump the following nginx-related variables in the Dockerfile:
   1. NGINX_VERSION
   2. NJS_VERSION
   3. NGINX_UNPRIVILEGED_COMMIT_HASH

Update nginx that hubble-ui depends on as follows.

1. Pick a commit hash from https://github.com/nginxinc/docker-nginx-unprivileged/commits/main/mainline/debian/Dockerfile
   • If NGINX_VERSION is 1.23.2, the commit hash is 85f846c6c5d121b2b750d71c31429d9686523da0 referencing the commit "Update mainline NGINX to 1.23.2"
   • You can find the corresponding NJS_VERSION value in the same commit
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.

# Check diff between v1.23.1 and v1.23.2
git diff 0b794b2bd54217ac3882680265c9426ae2edcbd6 85f846c6c5d121b2b750d71c31429d9686523da0 -- mainline/debian/Dockerfile

kube-metrics-adapter

1. Check the release page.
2. Update KMA_VERSION in Dockerfile.
3. Update TAG file.

kube-state-metrics

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/kubernetes/kube-state-metrics/blob/vX.Y.Z/Dockerfile
3. Update KUBE_STATE_METRICS_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update TAG files.

kube-storage-version-migrator

1. Check the release page.
2. Check the upstream build files. If there are any updates, update our Dockerfile.
   • https://github.com/kubernetes-sigs/kube-storage-version-migrator/blob/vX.Y.Z/Makefile
   • https://github.com/kubernetes-sigs/kube-storage-version-migrator/blob/vX.Y.Z/cmd/initializer/Dockerfile
   • https://github.com/kubernetes-sigs/kube-storage-version-migrator/blob/vX.Y.Z/cmd/migrator/Dockerfile
   • https://github.com/kubernetes-sigs/kube-storage-version-migrator/blob/vX.Y.Z/cmd/trigger/Dockerfile
3. Update MIGRATOR_VERSION in Dockerfile
4. Update TAG file.

kubernetes

1. Check the release page.
2. Update K8S_VERSION in Dockerfile.
3. Update image tag in README.md.
4. Update BRANCH and TAG files.

local-pv-provisioner

1. Update version variables in Makefile.

2. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.

3. Generate code and manifests.

   cd $GOPATH/src/github.com/cybozu/neco-containers/local-pv-provisioner
   make generate manifests
   # Commit, if there are any updated files.

4. Confirm build and test are green.

   make build test

5. Update image tag in local-pv-provisioner.yaml.

6. Update TAG file.

loki

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/grafana/loki/blob/vX.Y.Z/cmd/loki/Dockerfile
3. Update LOKI_VERSION in Dockerfile.
4. Update TAG file.

Note

Keep the version of promtail the same as that of loki.

machines-endpoints

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.

2. Confirm test is green.

   make test

3. Update image tag in machines-endpoints.yaml.

4. Update TAG file.

memcached

1. Check the release page.
2. Update MEMCACHED_VERSION in Dockerfile.
3. Update TAG file.

memcached_exporter

1. Check the release page.
2. Update MEMCACHED_EXPORTER_VERSION in Dockerfile.
3. Update BRANCH and TAG file.

meows-dctest-runner

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/cybozu-go/meows/blob/vX.Y.Z/Dockerfile
3. Update MEOWS_VERSION in Dockerfile.
4. Update GO_VERSION and PLACEMAT_VERSION in Dockerfile, if there are any updates.
   • GO_VERSION: https://github.com/cybozu/neco-containers/blob/main/golang-all
   • PLACEMAT_VERSION: https://github.com/cybozu-go/placemat/releases/latest
5. Update BRANCH and TAG files.

meows-neco-runner

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/cybozu-go/meows/blob/vX.Y.Z/Dockerfile
3. Update the Dockerfile to install the same tools as ubuntu-debug.
   • Also update GRPCURL_VERSION, if there are any changes.
   • https://github.com/cybozu/ubuntu-base/blob/main/22.04/ubuntu-debug/Dockerfile#L5
4. Update MEOWS_VERSION in Dockerfile.
5. Update BRANCH and TAG files.

moco-switchover-downtime-monitor

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.
2. Update TAG file.

opentelemetry-collector

opentelemetry-collector container consists of three repositories: opentelemetry-collector, opentelemetry-collector-contrib and opentelemetry-collector-releases

1. Check the release pages main contrib release.
2. Check the upstream Dockerfile and builder manifest. If there are any updates, update our Dockerfile.
   • https://github.com/open-telemetry/opentelemetry-collector-releases/blob/vX.Y.Z/distributions/otelcol/Dockerfile
   • https://github.com/open-telemetry/opentelemetry-collector-releases/blob/vX.Y.Z/distributions/otelcol/manifest.yaml
3. Update OTELCOL_VERSION in Dockerfile.
4. Update BRANCH and TAG files.

pause

1. Check the changelog.
   • https://github.com/kubernetes/kubernetes/blob/vX.Y.Z/build/pause/CHANGELOG.md
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/kubernetes/kubernetes/blob/vX.Y.Z/build/pause/Dockerfile
3. Update K8S_VERSION and PAUSE_VERSION in Dockerfile.
4. Update BRANCH and TAG files.

pod-delete-rate-limiter

1. Update the following version variables in Makefile.

   • CONTROLLER_TOOLS_VERSION
   • KUSTOMIZE_VERSION
   • ENVTEST_K8S_VERSION

2. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.

3. Generate code and manifests.

   cd $GOPATH/src/github.com/cybozu/neco-containers/pod-delete-rate-limiter
   make generate manifests
   # Commit, if there are any updated files.

4. Confirm build and test are green.

   make build test

5. Update TAG file.

pomerium

1. Check the release page and upgrade guide.

   • https://github.com/pomerium/pomerium/releases
   • https://www.pomerium.com/docs/core/upgrading

2. Check the diff of the Dockerfile.

   cd /path/to/pomerium
   git switch --detach v${NewVersion}
   git diff v${CurrentVersion} Dockerfile

3. Update Dockerfile.

   • Pomeruim version
   • Golang version
   • Node.js version

4. Update TAG file.

prometheus-adapter

1. Check the release page.
   • https://github.com/kubernetes-sigs/prometheus-adapter/releases
2. Update version variables in Dockerfile.
3. Update TAG file.

prometheus-config-reloader

This is a part of prometheus-operator. This is used by victoria-metrics operator too.

1. Check the latest release of prometheus-operator
2. Update version variable in Dockerfile.
3. Update TAG file.

promtail

Promtail contains two versions, one for promtail and the other for libsystemd. The promtail version should be the same with loki. The libsystemd version should be the same with the one running on the stable Flatcar OS.

1. Update LOKI_VERSION in Dockerfile.
2. Update SYSTEMD_VERSION in Dockerfile if needed.
3. Update TAG file.

promtail-debug

TBD

pushgateway

1. Check the release page.
   • https://github.com/prometheus/pushgateway/releases
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/prometheus/pushgateway/blob/vX.Y.Z/Dockerfile
3. Update version variables in Dockerfile.
4. Update TAG file.

redis

Note

This image is used by ArgoCD. So browse the following manifest and check the required version. If the manifest uses version a.b.c, we should use version a.b.d where d >= c. Don't use a newer minor version.

• https://github.com/argoproj/argo-cd/blob/vX.Y.Z/manifests/base/redis/argocd-redis-deployment.yaml

1. Check the release notes in the official site.
2. Check the Dockerfile in docker-library. If there are any updates, update our Dockerfile.
   • v7.0.x: https://github.com/docker-library/redis/blob/master/7.0/Dockerfile
3. Update REDIS_VERSION in Dockerfile.
4. Update BRANCH and TAG files.

registry

1. Check the release notes in the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/docker/distribution/blob/master/Dockerfile
3. Update REGISTRY_VERSION in Dockerfile.
4. Update BRANCH and TAG files.

rook

Note

If we update both Rook and Ceph, update Ceph image first, and then update Rook image.

Note

A specific version of rook depends on specific versions of csi sidecar containers listed below. Update these containers at the same time.

• cephcsi
• csi-attacher
• csi-node-driver-registrar
• csi-provisioner
• csi-resizer
• csi-snapshotter

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/rook/rook/blob/vX.Y.Z/images/ceph/Dockerfile
3. update build image tag in Dockerfile if necessary.
4. Update ROOK_VERSION in Dockerfile.
5. Update ceph image tag in Dockerfile.
6. Update BRANCH and TAG files.

Note

You may choose the latest docker image for the build, regardless of the upstream go version. The current go compiler builds with the language version and toolchain version based on the go version specified in the go.mod file. There is no need to use an older version of the image to match go.mod.

s3gw

Only the base image and module dependency should be updated.

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.
2. Update TAG by incrementing the patch revision, e.g. 1.0.1, 1.0.2, ...

sealed-secrets

1. Check the release page.
2. Check the upstream Dockerfile and compare with ours especially on the runtime stage. If there are any updates, update our Dockerfile.
   • https://github.com/bitnami-labs/sealed-secrets/blob/vX.Y.Z/docker/controller.Dockerfile
   • https://github.com/bitnami-labs/sealed-secrets/blob/vX.Y.Z/docker/kubeseal.Dockerfile
3. Update SEALED_SECRETS_VERSION in Dockerfile.
4. Update BRANCH and TAG files.

serf

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/hashicorp/serf/blob/vX.Y.Z/scripts/serf-builder/Dockerfile
3. Update SERF_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update BRANCH and TAG files.

squid

1. Check the latest stable version at http://www.squid-cache.org/Versions/
2. Check release notes if a new version is released.
   • e.g., http://www.squid-cache.org/Versions/vX/squid-X.Y-RELEASENOTES.html
3. Update SQUID_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update BRANCH and TAG files.

squid-exporter

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.
2. Update squid version in Makefile and e2e/pod.yaml if there are any updates.
3. Update TAG by incrementing the patch revision, e.g. 1.0.1, 1.0.2, ...

Note

The squid images should be updated at the same time as the squid-exporter image for consistency.

stakater/Reloader

1. Check the release page.
2. Check the upstream Dockerfile. If there are any updates, update our Dockerfile.
   • https://github.com/stakater/Reloader/blob/vX.Y.Z/Dockerfile
3. Update BRANCH and TAG files.

tcp-keepalive

TBD

teleport-node

1. Check the release page.
2. Run make -C teleport-node/ check-teleport-update and check the upstream Makefile and version.mk.
3. Update tools version in Dockerfile.
4. Update Dockerfile If there are any changes to the build method.
5. Update TELEPORT_VERSION in Dockerfile.
6. Update TAG files.

tempo

1. Check the release page.
2. Check the upstream Makefile and cmd/tempo//Dockerfile. If they have been updated significantly, update our Dockerfile.
   • https://github.com/grafana/tempo/blob/vX.Y.Z/Makefile
   • https://github.com/grafana/tempo/blob/vX.Y.Z/cmd/tempo/Dockerfile
3. Update TEMPO_VERSION in Dockerfile.
4. Update TAG file.

testhttpd

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.
2. Update BRANCH and TAG files.

trust-manager

1. Check the release page.
2. Update BRANCH and TAG files.

trust-packages

1. Check the release page.
2. Update TRUST_MANAGER_VERSION in Dockerfile.
3. Update TAG file.

ttypdb

Only the base image and module dependency should be updated.

1. Upgrade direct dependencies listed in go.mod. Use go get or your editor's function.
2. Update TAG by incrementing the patch revision, e.g. 1.0.1, 1.0.2, ...

unbound

1. Check the download page.
2. Run make update-root-hints.
3. Update UNBOUND_VERSION in Dockerfile.
4. Update image tag in README.md.
5. Update BRANCH and TAG files.

unbound_exporter

1. Check the release page
2. Update UNBOUND_EXPORTER_VERSION in Dockerfile.
3. Update BRANCH and TAG files.

vault

1. Check the release page and these notes:
   • https://www.vaultproject.io/docs/upgrading
   • https://www.vaultproject.io/docs/release-notes
2. Update VAULT_VERSION in Dockerfile.
3. Update image tag in README.md.
4. Update BRANCH and TAG files.

victoriametrics

1. Check the release page.
2. Check upstream Makefile and Dockerfile, and update our Dockerfile if needed.
   • https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vX.Y.Z/Makefile
   • https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vX.Y.Z/app/*/Makefile
   • https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vX.Y.Z/app/*/deployment/Dockerfile
   • https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vX.Y.Z-cluster/Makefile
   • https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vX.Y.Z-cluster/app/*/Makefile
   • https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vX.Y.Z-cluster/app/*/deployment/Dockerfile
3. Update VICTORIAMETRICS_SINGLE_VERSION and VICTORIAMETRICS_CLUSTER_VERSION in Dockerfile.
4. Update TAG file.

victoriametrics-operator

1. Check the release page.
2. Check upstream Makefile and Dockerfile, and update our Dockerfile if needed.
3. Update VICTORIAMETRICS_OPERATOR_VERSION in Dockerfile.
4. Update TAG file.