[WIP] Added mcad bool for Job submission with no cluster

[Bobbins228]

Issue link

Closes RHOAIENG-1055
linked to pytorch/torchx#822

What changes have been made

Added an mcad bool which defaults at False for Job submission.
When mcad is false we use the kueue_job torchx scheduler

Verification steps

• Follow the testing steps in this PR but stop after installation.
• Make a change to the pyproject.toml file
  • replace codeflare-torchx = "0.6.0.dev1" with torchx = {path = "/path/to/your/torchx/dist/torchx-0.7.0.dev0-py3-none-any.whl"}
• run poetry build and pip install --force-reinstall /path/to/your/dist/codeflare_sdk-0.0.0.dev0-py3-none-any.whl
• Run the jobs demo notebook with this configuration:

from codeflare_sdk.job.jobs import DDPJobDefinition
jobdef = DDPJobDefinition(
    name="mnistjob",
    script="mnist.py",
    # script="mnist_disconnected.py", # training script for disconnected environment
    scheduler_args={"namespace": "default", "local_queue": "YOUR LOCAL QUEUE"}, # ADD "priority_class":"kueue-priority-class-name" for testing priority
    j="1x1",
    gpu=0,
    cpu=1,
    max_retries=3,
    memMB=8000,
    image="quay.io/mcampbel/mnist-image:v0.0",
    mcad=False
)
job = jobdef.submit()

• From there you can test logs and status as well as cancel like the mcad version.

Note: If you are testing priority you need a workload priority class and the workload associated with the job should have the priority you set included.

Note 2: This PR is purely for testing the kueue job scheduler but will be converted to a "real" PR for adding Kueue Job submission when the codeflare-sdk torchx is synced with the upstream torchx repo

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from bobbins228. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment