LocalSession Enabling in Zookeeper

[Meet0861]

Pulsar version: 3.0.6
Context: we have global zk in 2 regions, R1->2 participants, 2 observers and R2->3 participants, 2 observers. Leader zk is in R2.
Now, lets say in some situation, the quorum is down and the R2 zks went to Read only mode.
Config: we have enabled the metadataStoreAllowReadOnlyOperations in the broker and got to know that localSessionEnabled should also be true as validation is added in this commit:apache/zookeeper@c47ef90 that global sessions dont work in RO mode.
Query: whats the impact of enabling the localSessions in Zookeeper from pulsar side? We read the documentation that local sessions to global session upgrades automatically once quorum is up. But is there any other impact of enabling this in normal situations?
Moreover, Even after enabling this, we have observed that broker makes RO connection with ZK, but still fetching metadata doesn't works and even lookups also dont work. is there any limitation on what sort of reads will work in RO mode?

[eolivelli]

fetching metadata doesn't works and even lookups also dont work. 

What do you mean? Do you have any errors to share?

AFAIK session upgrade is automatic.

The ZK is choosen automatically, if R2 is down another leader should be elected. But then you have only 2 participants in R1 and you lose quorum.

I can't remember what's the suggested deployment but with the current layout it is clear that if R2 is down then ZK cannot make progress

[Meet0861]

yea right, it loses the quorum. And the expectation when the quorum is lost and it is running in Read-Only mode, the read requests should be served.
Adding observations as per our testing based on the above mentioned zk setup where ZK is operating in RO mode:
pulsar v2.9.3 and Configstore v3.9.2(pulsar v3.0.7)
zookeeperStoreAllowReadOnlyOperations flag is not set in the brokers. Still existing reads/writes works. And few admin get calls also works. Though, from configstore we still see the exceptions like refusing the connection from not RO clients.

pulsar v3.0.7 and Configstore v3.9.2(pulsar v3.0.7)
metadataStoreAllowReadOnlyOperations flag is not set in the brokers. when zk lost the quorum and is in RO mode, brokers will not able to make connection with configstore and keep on restarting.
if we enable the metadataStoreAllowReadOnlyOperations in the broker and local session in the configstore, RO session establishment works and existing reads/writes also works but any admin call, lets say even get tenants fails with the keeperSessionExpired exception.
As broker sends a close session call to the configstore and fetching the admin policies fails.
Once the quorum is up again, session upgrades automatically and work as expected.

Now our concerns are,

• why in 2.9 it works without having the zookeeperStoreAllowReadOnlyOperations. What changed in 3.0.7 wrt the session establishment and handling the failures which led the current cluster in unstable state in case of quorum lost.
• why the get calls fails during the RO mode. is it expected?
• Is there anything which can be done so that new producer/consumer creation also works when configstore in RO? or does it make any write call internally to the configstore which we are not aware of.
• is there any negative impact of local sessions observed, from point of view

[eolivelli]

This looks like a bug, and we should report it as an issue. Please provide the version and all the pieces of information.

[Meet0861]

sure.
submitted this bug with all the details - #23838