[Bug] Ray Job submit fails when GCS is enabled

[XBeg9]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

export RAY_ADDRESS=http://172.27.0.143:8265 && ray job submit --verbose --working-dir job -- python script.py

RuntimeError: Request failed with status code 500: No available agent to submit job, please try again later..

Reproduction script

# script.py
import ray

@ray.remote
def hello_world():
    return "hello world"

# Automatically connect to the running Ray cluster.
ray.init()
print(ray.get(hello_world.remote()))

GCS is enabled on cluster (redis)

export RAY_ADDRESS=http://172.27.0.143:8265 && ray job submit --verbose --working-dir job -- python script.py

Anything else

Same cluster, but without GCS works as expected job is submitted.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!