Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option --keep-task to turn on/off scheduler persistence mode #23

Open
sharpener6 opened this issue Sep 27, 2024 · 0 comments
Open
Assignees
Labels
enhancement New feature or request

Comments

@sharpener6
Copy link
Collaborator

Currently, scaler scheduler is keeping the task object in the memory so if worker died, scheduler will allocate the task to other workers, but for memory efficiency, we will need have a mode that scheduler doesn't keep the task once sent to worker, so there are some behavior changes when this

keep task:

  • when task failed due to worker get disconnected, scheduler will reassign to another worker to do
  • when balancing tasks, scheduler just need task ids from busy worker and send to other workers

do not keep task:

  • when task failed due to worker get disconnected, scheduler will just return failed result to Client
  • when balancing tasks, because scheduler doesn't have task content at all, so it will ask busy workers to return not only task ids, but also task contents so it can reschedule to other workers
@sharpener6 sharpener6 self-assigned this Sep 27, 2024
@sharpener6 sharpener6 changed the title Add option --keep-task to turn on/off scheduler persistence mode FEATURE: Add option --keep-task to turn on/off scheduler persistence mode Sep 27, 2024
@sharpener6 sharpener6 changed the title FEATURE: Add option --keep-task to turn on/off scheduler persistence mode Add option --keep-task to turn on/off scheduler persistence mode Sep 27, 2024
@sharpener6 sharpener6 added the enhancement New feature or request label Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant