[KubeRay] support suspending worker groups in KubeRay autoscaler #49768
+30
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Resolves ray-project/kuberay#2666. ray-project/kuberay#2663 adds a new `suspend` field to the KubeRay worker group spec for suspending worker groups. A suspended worker group should be scaled to 0 and never be scaled up until the group is resumed. Since there is no similar functionality in the `available_node_types` definition, the best way to let the autoscaler know a worker group has been suspended is to set its max_workers to 0 as well as its min_workers. This PR makes the KubeRay autoscaling config producer produce a config with both max_workers and min_workers set to 0 if the worker group has been suspended. The autoscaler will periodically take the config and do its work. Signed-off-by: Rueian <[email protected]>
|
Hi @kevin85421, could you help review this? |
rueian
commented
Jan 11, 2025
| "min_workers": min_workers, | ||
| "max_workers": max_workers, | ||
| "min_workers": min_workers if not suspend else 0, | ||
| "max_workers": max_workers if not suspend else 0, |
There was a problem hiding this comment.
We need to let the autoscaler know that the suspended worker group should have no workers; otherwise, the autoscaler will keep trying to scale it up.
There was a problem hiding this comment.
Since the available_node_types definition does not provide similar functionality (suspending a node type), the best way to do that is to set its max_workers and min_workers to 0.
| @@ -312,6 +330,14 @@ def test_resource_quantity(input: str, output: int): | |||
| None, | |||
| id="autoscaler-options", | |||
| ), | |||
| pytest.param( | |||
| _get_ray_cr_with_groups_suspended(), | |||
| _get_autoscaling_config_with_groups_suspended(), | |||
There was a problem hiding this comment.
This tests both max_workers and min_workers are set to 0 in the generated autoscaling config.
rueian
force-pushed
the
kuberay-autoscaling-suspend
branch
from
afed960 to
fa119fd
Compare
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves ray-project/kuberay#2666.
ray-project/kuberay#2663 adds a new
suspendfield to the KubeRay worker group spec for suspending worker groups. A suspended worker group should be scaled to 0 and never be scaled up until the group is resumed.Since the
available_node_typesdefinition does not provide similar functionality (suspending a node type), the best way to let the auto scaler know a worker group has been suspended is to set its max_workers and min_workers to 0.max_workersandmin_workersof a suspended worker group set to 0 to inform the autoscaler that the suspended group should not have nodes. The autoscaler will periodically take the config and do its work.replicasvalues on the RayCluster CR. The suspended group will be scaled down by KubeRay operator instead.Related issue number
ray-project/kuberay#2666
ray-project/kuberay#2663
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.