Skip to content

Commit

Permalink
Fix metadata logging even when actors crash and processing_time is not
Browse files Browse the repository at this point in the history
updated.

In case the ray actors crash, the key `processing_time` may not be
updated to stats, so while rounding off this key, we may get an
exception which stops metadata from being written.

Signed-off-by: Shivdeep Singh <[email protected]>
  • Loading branch information
shivdeep-singh-ibm committed Oct 17, 2024
1 parent bd81cc9 commit fab1199
Showing 1 changed file with 3 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ def orchestrate(
if is_folder:
# folder transform
files = runtime.get_folders(data_access=data_access)
logger.info(f"Number of folders is {len(files)}") # Get files to process
logger.info(f"Number of folders is {len(files)}") # Get files to process
else:
files, profile, retries = data_access.get_files_to_process()
if len(files) == 0:
Expand Down Expand Up @@ -142,7 +142,8 @@ def orchestrate(
# Compute execution statistics
logger.debug("Computing execution stats")
stats = runtime.compute_execution_stats(ray.get(statistics.get_execution_stats.remote()))
stats["processing_time"] = round(stats["processing_time"], 3)
if "processing_time" in stats:
stats["processing_time"] = round(stats["processing_time"], 3)

# build and save metadata
logger.debug("Building job metadata")
Expand Down

0 comments on commit fab1199

Please sign in to comment.