Support nic-cluster-policy update

[abdallahyas]

Currently the network operator only support updating the components versions, any other value is ignored until either the component pod is restarted or the nic-cluster-policy is redeployed. The update needs to handle:

• Support deleting a component (delete the component daemonset) if the nic-cluster-policy was updated to not have that component. This can be more clear in case of helm based deployments, in which the value for a <component>.deploy was updated from true to false.
• Support changing the nic-cluster-policy configuration (other than the version) without the need to redeploy the nic-cluster-policy. This basically applies to the rdma-shared-dev-plugin in which case the pod should be restarted to take the new values, but can be extended to the other components as well like the whereabout IPAM plugin.
• Support updating both the mofed and the nv-peer-mem images without breaking the workloads.

[adrianchiris]

Another Important use-case for update would be the update of mofed , nv-peer-mem driver containers.
this bit is tricky as it will cause re-creation of all mellanox net-devices which may cause workloads to stop working.
as this happens at cluster scale we need to pay attention on how we would like to perform this.

[adrianchiris]

Item 1 and item 3 are handled. only item 2 remains.

item 2 can be done by either:

1. restarting device plugin
2. adding watch logic in rdma shared device plugin (if config map update is reflected in pod mounts, need to validate)