[Bug] one of the created Ray actors die during docid transform

[sujee]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/universal/doc_id

What happened + What you expected to happen

Could be related #719

full error

(orchestrate pid=5751) created [Actor(RayTransformFileProcessor, 37cbb100b21c9012ce68444101000000), Actor(RayTransformFileProcessor, e063fda8d2ea6f56efc152e201000000)], alive [ActorState(actor_id='37cbb100b21c9012ce68444101000000', class_name='RayTransformFileProcessor', state='ALIVE', job_id='01000000', name='', node_id='08deddebac466e0100f6c1015c266849f52c8ff173595c8170cdee12', pid=5915, ray_namespace='aa965802-2b25-4b0b-83c8-3e939d83ab34', serialized_runtime_env=None, required_resources=None, death_cause=None, is_detached=None, placement_group_id=None, repr_name=None)]
(orchestrate pid=5751) Traceback (most recent call last):
(orchestrate pid=5751)   File "/usr/local/lib/python3.10/dist-packages/data_processing_ray/runtime/ray/transform_orchestrator.py", line 89, in orchestrate
(orchestrate pid=5751)     processors = RayUtils.create_actors(
(orchestrate pid=5751)   File "/usr/local/lib/python3.10/dist-packages/data_processing_ray/runtime/ray/ray_utils.py", line 121, in create_actors
(orchestrate pid=5751)     raise UnrecoverableException(f"out of {len(actors)} created actors only {len(alive)} alive")
(orchestrate pid=5751) data_processing.utils.unrecoverable.UnrecoverableException: out of 2 created actors only 1 alive
(orchestrate pid=5751) 05:26:08 ERROR - Exception during execution out of 2 created actors only 1 alive: None
(orchestrate pid=5751) Traceback (most recent call last):
(orchestrate pid=5751)   File "/usr/local/lib/python3.10/dist-packages/data_processing_ray/runtime/ray/transform_orchestrator.py", line 138, in orchestrate
(orchestrate pid=5751)     stats["processing_time"] = round(stats["processing_time"], 3)
(orchestrate pid=5751) KeyError: 'processing_time'
(orchestrate pid=5751) 05:26:08 ERROR - Exception during execution 'processing_time': None
05:26:18 INFO - Completed execution in 2.309 min, execution result 1
INFO:data_processing_ray.runtime.ray.transform_launcher:Completed execution in 2.309 min, execution result 1

Reproduction script

code

colab link

error happens on step 5.2

Works with release 0.2.1

pip install  --default-timeout=100  \
        data-prep-toolkit-transforms==0.2.1 \
        data-prep-toolkit-transforms-ray==0.2.1 \
        deepsearch-toolkit

Fails with release 0.2.2.dev1

pip install  --default-timeout=100  \
        data-prep-toolkit[ray]==0.2.2.dev1 \
        data-prep-toolkit-transforms[ray,all]==0.2.2.dev1 \
        deepsearch-toolkit

Anything else

No response

OS

Other

Python

Other

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[daw3rd]

Based on the logged exception this seems to be a duplicate of #719 and is fixed in PR #721 that has been merged.