workflow pod is missing

[CyrilJacobPart]

Hello,

I can see arbitrary workflow pod is missing then pod deleted on a pod that is not deleted but cannot figure out from the log what is causing the issue.
I am assuming it is not deleted as I was expecting to see any call to io.k8s.core.v1.pods.delete on those pods but I can't.

time="2024-08-08T03:04:04.232Z" level=info msg="cleaning up pod" action=labelPodCompleted key=lynx/analytics-adhoc-activity-clm-template-1723086000-dbt-template-78707496/labelPodCompleted

time="2024-08-08T03:04:14.133Z"  level=info msg="**Workflow pod is missing**" namespace=lynx nodeName="analytics-adhoc-activity-clm-template-1723086000[0].dbt-build" nodePhase=Running recentlyStarted=false workflow=analytics-adhoc-activity-clm-template-1723086000

time="2024-08-08T03:04:14.133Z" level=info msg="node analytics-adhoc-activity-clm-template-1723086000-78707496 message: **pod deleted**" namespace=lynx workflow=analytics-adhoc-activity-clm-template-1723086000

time="2024-08-08T03:04:14.133Z" level=info msg="node analytics-adhoc-activity-clm-template-1723086000-78707496 p**hase Running -> Error**" namespace=lynx workflow=analytics-adhoc-activity-clm-template-1723086000