Handle failed json parsing events gracefully

[StrongestNumber9]

Can't continue as syntax for the event was invalid: com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 path $

Most likely caused by minikube rotating logs and file reading library gives invalid message

[StrongestNumber9]

Caught some:

24.05.2023 17:05:26.403 [INFO] [com.teragrep.k8s_01.K8SConsumer] [pool-2-thread-1] [0faa7370-a34c-485b-9087-e471f25e54fa] Invalid syntax message: g":"hi\n","stream":"stdout","time":"2023-05-24T17:05:23.650733687Z"}

24.05.2023 17:05:50.552 [INFO] [com.teragrep.k8s_01.K8SConsumer] [pool-2-thread-6] [5bbc112c-7f03-4e5e-9f66-3886c3cf476d] Invalid syntax message: 0.270350176Z"}

24.05.2023 17:05:50.547 [INFO] [com.teragrep.k8s_01.K8SConsumer] [pool-2-thread-5] [82d789a0-6dc5-438e-9c19-8b194e8fd408] Invalid syntax message: :"hi\n","stream":"stdout","time":"2023-05-24T17:05:48.867006088Z"}

24.05.2023 17:05:50.544 [INFO] [com.teragrep.k8s_01.K8SConsumer] [pool-2-thread-4] [a22e4888-efd5-4861-af39-269b3766ebfd] Invalid syntax message: t","time":"2023-05-24T17:05:49.389827802Z"}

Seems like there is a race condition between minikube truncating the logs and rlo_13 reading it:

root@minikube:~# while true; do wc -l /var/log/pods/default_first-pod_bc080c24-7f6b-4fab-9a0c-3fc4699b0390/first-pod-1/0.log; sleep 1; done;
1344463 /var/log/pods/default_first-pod_bc080c24-7f6b-4fab-9a0c-3fc4699b0390/first-pod-1/0.log
498077 /var/log/pods/default_first-pod_bc080c24-7f6b-4fab-9a0c-3fc4699b0390/first-pod-1/0.log
974348 /var/log/pods/default_first-pod_bc080c24-7f6b-4fab-9a0c-3fc4699b0390/first-pod-1/0.log
78241 /var/log/pods/default_first-pod_bc080c24-7f6b-4fab-9a0c-3fc4699b0390/first-pod-1/0.log
568624 /var/log/pods/default_first-pod_bc080c24-7f6b-4fab-9a0c-3fc4699b0390/first-pod-1/0.log
1030639 /var/log/pods/default_first-pod_bc080c24-7f6b-4fab-9a0c-3fc4699b0390/first-pod-1/0.log
80318 /var/log/pods/default_first-pod_bc080c24-7f6b-4fab-9a0c-3fc4699b0390/first-pod-1/0.log

Seems like minikube is removing the file and replacing it with a new one:

# while true; do grep "^[^{]" $(realpath /var/log/containers/first-pod_default_first-pod-1-34f13cf05ce007abfa5266e530bb224b6658652939e857e225b79f0960812962.log); sleep 1; done;
grep: /var/lib/docker/containers/34f13cf05ce007abfa5266e530bb224b6658652939e857e225b79f0960812962/34f13cf05ce007abfa5266e530bb224b6658652939e857e225b79f0960812962-json.log: No such file or directory
grep: /var/lib/docker/containers/34f13cf05ce007abfa5266e530bb224b6658652939e857e225b79f0960812962/34f13cf05ce007abfa5266e530bb224b6658652939e857e225b79f0960812962-json.log: No such file or directory
grep: /var/lib/docker/containers/34f13cf05ce007abfa5266e530bb224b6658652939e857e225b79f0960812962/34f13cf05ce007abfa5266e530bb224b6658652939e857e225b79f0960812962-json.log: No such file or directory

[StrongestNumber9]

Might be very well related to teragrep/rlo_13#6

[kortemik]

minikube does not truncate files

[kortemik]

minikube uses json-file logging driver which uses

https://github.com/moby/moby/blob/master/daemon/logger/loggerutils/logfile.go#L272-L291

os.Rename