Lightweight Python components

[RobbeSneyders]

The goal of this issue is to let user define custom components in Python only instead of having to build a complete docker component with component spec.

---

Original description:

The Dockerfile of most components is identical, but the user still needs to provide the whole component structure to create a component, which introduces unnecessary complexity.

I see two options to do this:

• Built the custom user code on top of a base image and use it like a docker component.
• Run the base image and provide the custom user code as an argument. Any additional dependencies are installed at runtime.

In both cases, we could make the base image configurable by the user.

The first option has some downsides:

• We need access to a user container registry to push to
• The base image would still need to be pulled before building and then pushing, which can take some time

The second option also has some downsides:

• Installs need to happen every time at runtime.
• It might be hard to pass complex code structures via argument.

We could have a look at how Kubeflow's lightweight Python components are implemented.

We might also be able to generate the component spec from the Python code:

• Arguments could be inferred by inspecting the __init__ method.
• If we find a way to typehint dataframes, we could also infer the consumes and produces section. Tools like Pandera could be helpful in this.

---

Tasks

Beta Give feedback

1. Support applying Lightweight Python components in Pipeline SDK #750
   Core +1
   
2. Infer the arguments based on component __init__ arguments #751
   Core +1
   
3. Validate consumes and infer produces for Lightweight Python components #752
   Core +1
   
4. Support Lightweight Python components in Docker runner #753
   Core +1
   
5. Support Lightweight Python components in Vertex & KfP runners #754
   Core +1
   
6. Support Lightweight Python components in Sagemaker runner #755
   Core +1
   
7. Add CI/CD to build default Fondant base image #782
   CI/CD +1
   
8. Add validation to lightweight python component decorator #783
   Core +1
   
9. Enable mapping of field names with consumes argument for lightweight components #785
   Core +1
   
10. Test caching for Python components and update if needed #787
    Core +1
    
11. Document lightweight python components #799
    4 of 4
    documentation +1
    

[GeorgesLorre]

We will need to be strict with the lightweight and clearly decide what it can and can't do. Otherwise it will become very complex very fast. We also need to think about how easy it would be to convert (for a user) a lightweight component to a real custom component (and a reusable one).

I'm a bit concerned these will become the first class way of doing things which makes sense since it the easiest but it will unload a lot of complexity to the fondant side.

[RobbeSneyders]

I think it will indeed become the default way to run custom code, but I believe we need to make that easier to make Fondant more useful. Moving from a Lightweight Python component to a Docker component should only be needed when a user wants to make the component reusable or runs into a limitation of the Lightweight Python component.

I would remove as many limitations as possible, but can see the following:

• Everything should be encapsulated in the Component class. Probably even non-Fondant imports (Fondant imports could already be needed to define the class, we can add those ourselves like kfp does.
• Only support Python dependencies. If a component needs non-Python dependencies, it needs to be dockerized. This will allow us to be more flexible about which base docker image to run the component on or even run it outside of docker (eg. with a VenvRunner).

Moving from a Lightweight Python component to a reusable Docker component should be quite straightforward since the Python code can stay the same. The user will need to add a component specification and Dockerfile, and build it and make it available for remote runners.

I think the additional complexity on the Fondant side can be limited and best effort. Eg. we can try to infer types so the user doesn't need to define the consumes and produces arguments, but they can still do that as a fallback when inference does not succeed for some reason.