Proposed Changes to Fix Concurrency Issues

[kundadebdatta]

Pull Request Template

Description

1. Validate only the Unhealthy replicas, and remove the Unknown ones from validation.
2. Add a mechanism to get hold of a connection health state, to detect any established connection before attempting to open new connections.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

Closing issues

To automatically close an issue: closes #IssueNumber

[github-actions]

Please follow the required format: "[Internal] Category: (Adds|Fixes|Refactors|Removes) Description"

Internal should be used for PRs that have no customer impact. This flag is used to help generate the changelog to know which PRs should be included. Examples:
Diagnostics: Adds GetElapsedClientLatency to CosmosDiagnostics
PartitionKey: Fixes null reference when using default(PartitionKey)
[v4] Client Encryption: Refactors code to external project
[Internal] Query: Adds code generator for CosmosNumbers for easy additions in the future.

[kundadebdatta]

We might need to invest some time to check the perf of the dictionary creation and maintaining for a large partition. Ideally the size should be constant and proportional to the number of partitions.