Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removing Beanstalk from teuthology #1650

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Commits on Oct 8, 2024

  1. Remove usage of beanstalk from teuthology

    The following changes support the removal of Beanstalk from Teuthology.
    In place of Beanstalk, we will now be using Paddles for queue management in Teuthology.
    This PR has the corresponding changes for the paddles PR: https://github.com/ceph/paddles/pull/94/files.
    
    The changes include:
    1. Removing all beanstalk related code
    2. Teuthology scheduler and dispatcher using Paddles queue for scheduling and dispatching jobs
    3. Adding support for Paddles queue management
    4. Additional functionality of being able to change the priority of Teuthology jobs in the queued state in the teuthology-queue command
    
    Signed-off-by: Aishwarya Mathuria <[email protected]>
    amathuria committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    9d97b17 View commit details
    Browse the repository at this point in the history
  2. Modify teuthology worker unit test and fix flake8 errors

    Previously the teuthology worker unit test used beanstalk, the test will
    be using Paddles now.
    
    Signed-off-by: Aishwarya Mathuria <[email protected]>
    amathuria committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    99e5ca6 View commit details
    Browse the repository at this point in the history
  3. Add retry for paddles calls and modify pause queue command

    1. Add retry loop for the paddles calls.
    2. Add run name as a parameter for updating priority of jobs in paddles.
    3. Modify the pause queue command to run on server side with an optional pause duration parameter.
    
    Signed-off-by: Aishwarya Mathuria <[email protected]>
    amathuria committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    72d4408 View commit details
    Browse the repository at this point in the history
  4. Add new test cases for dispatcher and scheduler

    Signed-off-by: Aishwarya Mathuria <[email protected]>
    amathuria committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    0cf9a5e View commit details
    Browse the repository at this point in the history
  5. Add beanstalk as a possible queue backend for Teuthology Jobs along w…

    …ith Paddles
    
    With the use of the --queue-backend argument the user can specify which backend(paddles/beanstalk) they would like to use for maintaining the teuthology Jobs queue.
    In order to avoid overlapping Job IDs, when a job is being scheduled in beanstalk it is also written to paddles which returns a unique ID.
    This is the ID teuthology will treat as the Job ID throughout the run of the job.
    
    To differentiate between the 2 queue backends, the teuthology-queue command has been split into teuthology-paddles-queue command and teuthology-beanstalk-queue command.
    
    Signed-off-by: Aishwarya Mathuria <[email protected]>
    amathuria committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    e9122c6 View commit details
    Browse the repository at this point in the history
  6. scripts/queue: common teuthology-queue command for paddles and beanst…

    …alk queue
    
    Signed-off-by: Aishwarya Mathuria <[email protected]>
    amathuria committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    74bd369 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. teuthology/queue: Single command for queue operations

    Makes the same teuthology-queue commands work regardless of the queue backend, Paddles or Beanstalk.
    
    Signed-off-by: Aishwarya Mathuria <[email protected]>
    amathuria committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    be34af2 View commit details
    Browse the repository at this point in the history