Should it be possible to set up log shipping to elastic via fluentd?

[e-nikolov]

Not sure how to ask my question in a better way, basically has anyone managed to set up fluentd with k3s for log shipping?

[e-nikolov]

I'll answer my own question in case others attempt the same. It is possible with https://github.com/fluent/fluentd-kubernetes-daemonset, but there were a number of gotchas.

The fluentd logs started filling up with infinite backslashes ("\\\\\\...."). Issues fluent/fluentd-kubernetes-daemonset#186 and openshift/origin-aggregated-logging#1423 (comment) suggested that fluentd was processing its own logs which resulted in a recursive behaviour. To solve this, I had to add exclude_path ["/var/log/containers/fluentd*"] to the source section of its config file.

The second gotcha was that unlike docker, containerd doesn't produce logs in a json format, but a custom one, e.g. 2018-06-26T01:37:58.737599779Z stderr F <log message goes here>. In order to solve this, I replaced the @type json with

     <parse>
        @type regexp
        expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
        time_format %Y-%m-%dT%H:%M:%S.%NZ
     </parse>

Additionally I had elastic search in the same docker-compose.yml as k3s and I could access it from the k3s container via its service name (e.g. ping elasticsearch worked), but from the fluentd pod I could only access it via its IP address within docker (which can change).

To fix this I used a combination of k3s agent --resolv-conf /etc/resolv.conf for k3s to inherit the /etc/resolv.conf of its docker container and adding hostNetwork: true to the fluentd pod spec for it to inherit the /etc/resolv.conf of k3s.

[mateeyowt]

Thanks a lot @e-nikolov! You saved my ass.

[cjellick]

I think we can document this better in the future

[dereknola]

As docker in no longer the CRI layer for K3s, containerd is, this Issue is very OOD and no docs change is necessary. Users should be able to get fluentd working with K3s by following https://docs.fluentd.org/v/0.12/articles/kubernetes-fluentd and fluent/fluentd-kubernetes-daemonset#412