[Clusters] Cluster setup fails with pydantic 1.10.20

[georgeholt1]

What happened + What you expected to happen

Running ray up cluster-config.yaml fails when setting up a cluster on AWS.

Relevant part of the logs:

  [7/7] Starting the Ray runtime
Traceback (most recent call last):
  File "/home/ubuntu/aloha/.venv/bin/ray", line 5, in <module>
    from ray.scripts.scripts import main
  File "/home/ubuntu/aloha/.venv/lib/python3.12/site-packages/ray/scripts/scripts.py", line 2636, in <module>
    from ray.util.state.state_cli import (
  File "/home/ubuntu/aloha/.venv/lib/python3.12/site-packages/ray/util/state/__init__.py", line 1, in <module>
    from ray.util.state.api import (
  File "/home/ubuntu/aloha/.venv/lib/python3.12/site-packages/ray/util/state/api.py", line 18, in <module>
    from ray.util.state.common import (
  File "/home/ubuntu/aloha/.venv/lib/python3.12/site-packages/ray/util/state/common.py", line 944, in <module>
    @dataclass(init=not IS_PYDANTIC_2)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/dataclasses.py", line 232, in pydantic.dataclasses.dataclass.wrap
    float,
  File "pydantic/main.py", line 813, in pydantic.main.BaseModel.__try_update_forward_refs__
  File "pydantic/typing.py", line 566, in pydantic.typing.update_model_forward_refs
    raise TypeError(f"{self} is not subscriptable")
  File "pydantic/typing.py", line 532, in pydantic.typing.update_field_forward_refs
    class Any(metaclass=_AnyMeta):
  File "pydantic/typing.py", line 75, in pydantic.typing.evaluate_forwardref
    'Iterable',
TypeError: ForwardRef._evaluate() got an unexpected keyword argument 'type_params'

The cluster config I'm using is similar to the official example except I am not using the provided docker image. Instead, I am installing Ray and the dependencies myself. Part of the dependencies is pydantic<2 as recommended in the python requirements.txt file.

This has worked for the last few months until yesterday when pydantic 1.10.20 was released.

Setting pydantic=1.10.19 results in no issues when setting up the cluster.

Versions / Dependencies

ray 2.40.0
pydantic 1.10.20

Reproduction script

ray up cluster-config.yaml

Issue Severity

None