Bug: Restarting builder and claudie-operator during a cluster destroy causes a hang in the workflow

[bernardhalas]

Current Behaviour

When wiping the whole configuration (kubectl delete inputmanifest ...), there's a risk of ending up in a hanging state if builder is restarted (force-deleted or crashed) during the process.

Expected Behaviour

The expectation is that the cluster destroy finishes successfully.

Steps To Reproduce

1. Trigger a cluster deletion.
2. After some time, restart the builder and claudie-operator pods.
3. See that the workflow will remain stuck in the destroy phase.

Anything else to note

Not sure whether this has been caused by the builder or claudie-operator restart. I'll try to narrow this time if my time permits.

[Despire]

@bernardhalas have you forced delete the pods ? 'cause when you delete them, they'll finish whatever workflow is currently running, so in theory they should not hang.

[JKBGIT1]

Let's wait for 2 hours to if the workflow gets picked up. After that capture the relevant logs.

[bernardhalas]

Closing as non-reproducible.