-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document how to use rust UDF extensions of datafusion-python #792
Comments
Based on excellent feedback on the discord server, I have a minimum demonstrable of doing this that does not require any code changes in datafusion-python. I'm going to document what I have. |
https://github.com/timsaucer/tuple_filter_example/blob/main/src/lib.rs I think my code must be making a copy of the data because it is slower than I would expect. I will continue to investigate. |
Try using Running with
Re-running with
NOTE: I had to convert all
|
Awesome! Thank you! |
If this PR goes in, we can possibly just update the user documentation with a minimal example and link to the blog post: apache/datafusion-site#17 |
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Frequently when we are working on a set of new features or data analytics we find recurring patterns in our code. We will typically abstract these into a python package to share across the team. There are times where we must use UDFs to process some data that are non-performant since we have to convert the data to python objects when the existing pyarrow operations are insufficient for our needs.
I would like to be able to take these UDFs and author them in rust and use these user defined functions in python. Currently there is a blocker in that data cannot be shared from one rust crate to another. PyO3/pyo3#1444 is a tracking issue that discusses this in more detail.
Describe the solution you'd like
It appears at first glance that
PyCapsule
approach is the best way forward based on examples in pyo3_arrow and rust-numpy.I have a minimal example that I will try to put up later this week into a repo to demonstrate what I would like to accomplish. Basically, I should be able to define a UDF in rust so that I can make it highly performant, add a
pyo3
wrapper function around it, and use this python module alongside the existingdatafusion-python
.Describe alternatives you've considered
I'm uncertain at this point about other options besides a very painful reexport of the entire datafusion python module.
Additional context
I will try to push up my minimal demonstration repo later this week.
The text was updated successfully, but these errors were encountered: