draft: fixes fencing and hot-reload in 2.11+ #29
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
August 20, 2023 14:05
* due to built-in retries on non-200 responses config.etcd:list() takes N*timeout to complete instead of just timeout. Fencing is time-bound algorithm, so listings should not overcommit fencing_timeout
* this patch introduces basic test-suite for most common topologies: etcd.cluster.master and etcd.instance.single Test-suite starts 1/2/3 tarantool instances against 3 etcd nodes and checks that configuration is valid. Test-suite also checks that config is reloadable. For now test suite is running on 1.10.15, 2.8.4, 2.10.6, 2.11.0 and 2.11.1
|
fixes #27 |
added 2 commits
September 13, 2023 09:43
* feat: support bootstrap_strategy (2.11)
* feat: derive load_cfg/default_cfg
* fix: ensure options of box.cfg always cleared before passing
to box.cfg (or wrap_box_cfg, or boxcfg)
* feat: added annotations for config and config.etcd
* feat: use module-aware logger (2.11 feature)
* feat: expose load_cfg/{default_cfg,...} to config._load_cfg
ochaton
commented
Oct 3, 2023
* This method is needed for autofailover mechanisms mainly to enforce
read_only=true during loading phase of tarantool.
Before this patch race condition existed between autofailover and
moonlibs/config recovering behaviour.
If master crashes but fastly restarts then it initiates long running
loading phase. Master recovers as read_only=true but after returning
from box.cfg moonlibs/config retrieves config from ETCD and rechecks
read_only option.
The race happens when autofailover changes configuration in ETCD, but
master just in time returns from loading phase and applies oldest
configuration. This leads cluster to split-brain.
With method config.enforce_ro it is possible for external coordinator
firstly enforce_ro on loading leader and receive approval that leader
will not be promoted to rw until next reload configuration.
tarantool is enforcable to be ro only when all of the following conditions
are met:
1) Tarantool is recovering from snapshot (it was already
bootstrapped)
2) Client's code do not override box.cfg with passing args.boxcfg
3) args.tidy_load is enabled (default, but can be overriden by
client)
4) config uses ETCD to retreive topology.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
{mt_call, reload_cfg, reconfig_modules}etcd_loadcall to simplify investigations of issues related to "smthing happened to tarantool-etcd"fencing_timeoutis exhausteddeadlineand is used to adjust request-timeout in listings@Mons , please have a look. And do not merge yet :)