Missing error handling in benchmark_sdxl_rocm.py

[ScottTodd]

On iree-org/iree#17847, compilation failed and the benchmark job using iree_tests/benchmarks/sdxl/benchmark_sdxl_rocm.py at 3603a45 did not handle that gracefully:

https://github.com/iree-org/iree/actions/runs/9874435266/job/27269408927#step:16:46

INFO     root:benchmark_sdxl_rocm.py:31 Command failed with error: b''
INFO     root:benchmark_sdxl_rocm.py:161 Running SDXL ROCm benchmark failed. Exiting
INFO     root:benchmark_sdxl_rocm.py:179 E2E Benchmark Time: None ms (golden time 320.0 ms)

...

>       check.less_equal(benchmark_e2e_mean_time, goldentime_rocm_e2e, "SDXL e2e benchmark time should not regress")
E       TypeError: '<=' not supported between instances of 'NoneType' and 'float'

SHARK-TestSuite/iree_tests/benchmarks/sdxl/benchmark_sdxl_rocm.py:298: TypeError

[ScottTodd]

Traced this a bit.

We only log stderr on failure here, but we still return stdout:

SHARK-TestSuite/iree_tests/benchmarks/sdxl/benchmark_sdxl_rocm.py

Lines 23 to 32 in 3603a45

	def run_iree_command(args: Sequence[str] = ()):
	command = "Exec:", " ".join(args)
	logging.getLogger().info(command)
	proc = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False)
	stdout_v, stderr_v, = proc.stdout, proc.stderr
	return_code = proc.returncode
	if return_code == 0:
	return 0, proc.stdout
	logging.getLogger().info(f"Command failed with error: {proc.stderr}")
	return 1, proc.stdout

We get that stdout output here and pass it to job_summary_process:

SHARK-TestSuite/iree_tests/benchmarks/sdxl/benchmark_sdxl_rocm.py

Lines 174 to 176 in 3603a45

	# e2e benchmark
	ret_value, output = run_sdxl_rocm_benchmark(rocm_chip, gpu_number)
	benchmark_e2e_mean_time = job_summary_process(ret_value, output)

The stdout output is then ignored in job_summary_process if the ret value is 1:

SHARK-TestSuite/iree_tests/benchmarks/sdxl/benchmark_sdxl_rocm.py

Lines 159 to 167 in 3603a45

	def job_summary_process(ret_value, output):
	if ret_value == 1:
	logging.getLogger().info("Running SDXL ROCm benchmark failed. Exiting")
	return
	bench_lines = output.decode().split("\n")[3:]
	benchmark_results = decode_output(bench_lines)
	logging.getLogger().info(benchmark_results)
	benchmark_mean_time = float(benchmark_results[10].time.split()[0])
	return benchmark_mean_time

[ScottTodd]

New coverage with pytest prior to the benchmark script also helps here.

[ScottTodd]

Landed a fix in IREE. Can copy it to this repo as well or just call this fixed.