Validate consumes
and infer produces
for Lightweight Python components
#752
Open
Labels
Core
Core framework
When the user uses Lightweight Python components (#558) we want to get any information we currently get from the component spec from the provided Python code.
For the
consumes
section, we can assume it matches the schema of thedataset
the operation is applied to, possibly altered by theconsumes
argument passed to theapply
method.For the
produces
section, the user can either provide a schema via theproduces
argument on theapply
method, or we can try to infer it by simulating thetransform
function. We could do this by generating dummy data based on theconsumes
schema, and applying thetransform
method on it.This only makes sense for Transform components since we always expect the user to provide a
produces
schema for a Read component, and a Write component doesn't produce anything.Inferring the
produces
schema by simulation would also validate theconsumes
schema if it succeeds. It doesn't invalidate it when failing though, since there can be multiple reasons for a failed simulation: either theconsumes
schema is incorrect, there's a bug in the component, or a bug in the dummy data generation.The text was updated successfully, but these errors were encountered: