Set dqlite failure-domain based on the k8s node AZ label #989
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
petrutlucian94
changed the title
This commit listens for node label changes and configures the dqlite failure domain based on the availability zone node label. Dqlite adjusts the cluster node promotion order based on the failure domain, attempting to spread the quorum members over different failure domains.
petrutlucian94
force-pushed
the
lpetrut/dqlite-az
branch
from
d76868a to
44374e1
Compare
petrutlucian94
changed the title
|
I can add some tests after the current implementation is approved. |
petrutlucian94
commented
Jan 21, 2025
bschimke95
reviewed
Jan 21, 2025
There was a problem hiding this comment.
Great work @petrutlucian94 - did a first round, mainly focused on overall structure.
* move some of the logic to the node label controller
* if just one of the db configs was updated, avoid restarting the
other one
* allow clearing the failure domain
* add node label controller tests * move some of the dqlite utils to separate functions that can be reused by the tests
|
@bschimke95 thanks for reviewing this PR, I've addressed the comments and submitted some unit tests. |
We added the getNewK8sClientWithRetries helper to avoid code duplication among k8sd controllers but missed the fact that some controllers need a node (kubelet) client while others need an admin client. We'll add a boolean to request the expected client.
We're adding a test that sets k8s node availability zone labels, ensuring that the failure domain is applied correctly and that the cluster remains functional.
berkayoz
approved these changes
Jan 24, 2025
"until" is expected to receive a callable, we need to pass a lambda to ensure that the check is correctly retried in case of exceptions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit listens for node label changes and configures the dqlite failure domain based on the availability zone node label.
Dqlite adjusts the cluster node promotion order based on the failure domain, attempting to spread the quorum members over different failure domains.