Use native sidecar containers

[tronghn]

Kubernetes natively supports sidecar containers, which has been available in beta since Kubernetes 1.29. Slated for GA in 1.33.
Upstream issue tracking: kubernetes/enhancements#753

Depending on an Application's configuration, Naiserator may inject long-lived containers that run alongside the main application container.

Examples include:

• Cloud SQL proxy - auth proxy for Cloud SQL
• Elector - provides an API for leader election
• Fluentbit - ships secure logs (already uses native sidecar)
• Wonderwall - reverse proxy for user authentication

We should update the remaining containers to also use the new spec.

Seems like there are still some issues to be resolved. Keep these in consideration if we decide to update the remaining containers before GA.

[kimtore]

Do we want to wait for GA in 1.33 or try to integrate changes now?

@tronghn, what do you think after reading that issue?

[tronghn]

kubernetes/kubernetes#127958 would affect Wonderwall at least, and it was only recently fixed. We could work around this by using numeric ports instead, though I'm not sure how this plays out with our logging and prometheus stack.

On the other hand, we already have one sidecar using this feature and it seems to be stable enough for that use case.

We could try moving these one by one and iron out any problems on the way, though I think we'd benefit from just waiting until 1.32 or GA. The current set up with "normal" containers has been working fine without any major issues.

[sklirg]

kubernetes/kubernetes#127958 would affect Wonderwall at least, and it was only recently fixed. We could work around this by using numeric ports instead, though I'm not sure how this plays out with our logging and prometheus stack.

It's also possible to work around it by injecting the (named) port in "one of the main containers" instead of in the sidecar/initContainer, as workaround number 2 demonstrates. Then, in the future, when this is fixed (it's targeting Kuberetes v1.32), the injection can move to the initContainer instead. More hacky, but retains the named port functionality.