Verify CP Subsystem usage on Kubernetes

[leszko]

No description provided.

[leszko]

I checked and currently, Hazelcast CP Subsystem does not work correctly in Kubernetes. This is because the assumption of CP Subsystem is that the Hazelcast cluster is stable (stable IP addresses) and if any member is restarted it should be manually removed from the cluster as described in Hazelcast Reference Manual: CP Subsystem’s Fault Tolerance Capabilities. In Kubernetes however, restarting PODs is an automatic (and expected) operation.

I tested what happens with the enabled CP Subsystem when the rolling upgrade is performed. The member fails with the following logs and CP Sybsystem Group never recovers.

[sancar]

Hi Rafal, Did you enable CP Persistence as I suggested here to the user hazelcast/hazelcast#15299 (comment) ?

[leszko]

I tried it now, and actually, yes, it solves the issue. I updated the doc here: #289

Additionally, we need to support it in the Helm Chart, created an issue here: hazelcast/charts#176