Support soft and hard affinity in nodeAntiAffinity

[tczhao]

Summary

What change needs making?

Setting nodeAntiAffinity prevents running steps on the same host when pod retries.
This is configured by patching requiredDuringSchedulingIgnoredDuringExecution filter out hostnode in the pod affinity.
However, we could encounter a situation where retry exhausts all the hostnodes.

We want this configuration to be more flexible,
e.g. supports preferredDuringSchedulingIgnoredDuringExecution
e.g. supports configuring maximum number of hosts excluded

---

Message from the maintainers:

Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.