Reload fluentX certificates when the TLS secrets changes

[aslafy-z]

Is your feature request related to a problem? Please describe.
I generate certificates for fluentbit and fluentd using cert-manager. When they are renewed, logging-operator does not rollout the resources. I found that fluentbit was not re-loading the TLS certificates by itself when the file mounted from the secret is updated (see fluent/fluent-bit#1678). Not sure about fluentd.

Describe the solution you'd like
Logging operator watches the secrets and start a rollout when it changes.

Describe alternatives you've considered
Use stackater/reloader to do so, but I think it's a bit hackish. Logging operator should manage the sts/ds by itself.

[aslafy-z]

I tried to use Reloader to start the rollouts, but it appears that there is no statefulsetAnnotations / daemonsetAnnotations field, which prevents the use of it.

# LoggingSpec
fluentbit:
  annotations:
    secret.reloader.stakater.com/reload: "tls-fluentbit"
fluentd:
  annotations:
    secret.reloader.stakater.com/reload: "tls-fluentd"

this adds annotations the statefulset / daemonset pod template, not in its own metadata.

[aslafy-z]

It looks like labels are propagated to the daemonset/statefulset and their pod template. It's not the case for annotations, which is understandable since it can cause issues.

https://github.com/banzaicloud/logging-operator/blob/master/pkg/resources/fluentbit/meta.go#L27
https://github.com/banzaicloud/logging-operator/blob/master/pkg/resources/fluentd/meta.go#L27

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[aslafy-z]

This should be solved to give a smooth experience with tls. Currently, the main workaround is to disable tls :/

[pepov]

Do other operators do this? I'm a bit reluctant to do this if stackater/reloader does a good job. I would try not to add complexity where it is not absolutely necessary. To be honest I would expect the logging agent be smart enough to handle this. The only thing that sounds reasonable to me to watch for changes and reload the agent inside the pod, if the agent supports that.
Please note that we are working on supporting an alternative (syslog-ng) along fluentbit to tail and forward logs and that could possibly solve this issue as well, since syslog-ng allows reloading it's configuration runtime.

[pepov]

Converting this to a discussion until we can settle with a proposal