[Bug] HAP kfp test failing

[touma-I]

Search before asking

• I searched the issues and found no similar issues.

Component

KFP workflows

What happened + What you expected to happen

KFP-test v1 and V2 are failing for KFP workflow without a clear indication of the source of the error. This occur as part of a PR that retriggered the workflow for ALL the transforms. They all passed except for this one.

#22 exporting to image
#22 exporting layers
#22 exporting layers 37.6s done
#22 writing image sha256:0087556f03d3b1ba79ab24771a6c73258f05ef6962832c15d671be2506c2c3fb done
#22 naming to docker.io/library/hap-ray:latest done
#22 DONE 37.6s
docker tag hap-ray:latest  quay.io/dataprep1/data-prep-kit/hap-ray:latest
make[4]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/universal/hap/ray'
rm -rf data-processing-lib-python
rm -rf data-processing-lib-ray
rm -rf python-transform 
make[3]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/universal/hap/ray'
make[3]: Entering directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/universal/hap/ray'
kind load docker-image quay.io/dataprep1/data-prep-kit/hap-ray:latest --name=dataprep
Image: "quay.io/dataprep1/data-prep-kit/hap-ray:latest" with ID "sha256:0087556f03d3b1ba79ab24771a6c73258f05ef6962832c15d671be2506c2c3fb" not yet present on node "dataprep-control-plane", loading...
Image: "quay.io/dataprep1/data-prep-kit/hap-ray:latest" with ID "sha256:0087556f03d3b1ba79ab24771a6c73258f05ef6962832c15d671be2506c2c3fb" not yet present on node "dataprep-worker2", loading...
Image: "quay.io/dataprep1/data-prep-kit/hap-ray:latest" with ID "sha256:0087556f03d3b1ba79ab24771a6c73258f05ef6962832c15d671be2506c2c3fb" not yet present on node "dataprep-worker", loading...

Reproduction script

Re-enable the kfp_ workflow for HAP and re-run CICD

Anything else

This problem is persistent even after restarting the test in travis. For now, the kfp_ folder was renamed/disable so we can proceed with other work until we find the reason for this problem.

OS

Ubuntu

Python

3.10.x

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[touma-I]

Image size is about 7.9GB with a large number of layers. Most likely unable to load in Travis VM.

[touma-I]

@revit13 @roytman What is the clip level for KFP_BLACK_LIST ? the hap ray image is 7.9GB and the workflow test is hanging in travis so I am assuming it is exceeding the maximum limit. Please confirm. Thanks

[revit13]

The size of the transformed images I observed, which were successfully tested on the Kind cluster, was around 2.5GB to 3GB. So I would say that HAP can be in the black list.