Restart NOT_RUNNING nodes in parallel

[tinaselenge]

So I have tested the following changes to address strimzi/proposals#103 (comment):

• If all controllers are in NOT_RUNNING state, including pure and combined controllers, then we restart them in parallel.
• All NOT_RUNNING nodes that have old revision will be handled in parallel as suggested but this includes nodes with any role, e.g. controller and broker nodes can get restarted at the same time. Do you think we should separate them by roles?
• When restarting nodes unresponsive to connections, in case there are more than one, they will be restarted one by one, in the order of pure controller, combined and broker.

[ShubhamRwt]

For the second pointer in the PR description, does it mean even the active controller or? Maybe rolling the broker nodes first and then the controller nodes?

[tinaselenge]

For the second pointer in the PR description, does it mean even the active controller or? Maybe rolling the broker nodes first and then the controller nodes?

So I think if the node is in NOT_RUNNING state, it shouldn't be the active controller. If it was previously, a new active controller should have been elected. If all nodes are in this state, that means there is no quorum or active controller at all. We could split the batch to be restarted into controllers and brokers batches, so that was my question if that's necessary or not. If we do that, I think the controller batch should be restarted first, since that's the order we have been following, as controllers would need to be fixed first in order for brokers to work.

[ShubhamRwt]

Right, sorry I confused that we were starting controller first, maybe feedback from Tom would be great too?