Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for executing Substrait plans #10

Open
devinjdangelo opened this issue Feb 21, 2024 · 2 comments
Open

Add support for executing Substrait plans #10

devinjdangelo opened this issue Feb 21, 2024 · 2 comments

Comments

@devinjdangelo
Copy link
Collaborator

The existing FlightSQLExecutor executes a sql string. We should add support for executing a serialized Substrait plan. In the longer term , it should be easier to support more advanced pushdown for engines which support consuming Substrait.

@devinjdangelo
Copy link
Collaborator Author

I plan to take a look at apache/datafusion#9299 upstream to get some more familiarity with how Datafusion serializes plans to substrait and what limitations it may have currently.

@backkem
Copy link
Collaborator

backkem commented Feb 21, 2024

In the current approach we're federating the logical plan. So, this would likely be done earlier then the error mentioned in that issue.

It should be fairly simple to add Substrait-based federation to the FlightSQL example. There is fairly fleshed out import and export code for logical plans. My idea was to, in the short term, just have a boolean to determine if SQL or Substrait should be sent. The alternative/eventual goal could be to build out the Flight SQL introspection commands to automate that further.

Adding support for federating (part of) physical plans is another avenue that could be explored in the future.

edmondop pushed a commit to edmondop/datafusion-federation that referenced this issue Jul 6, 2024
…uery when executing (datafusion-contrib#10)

* keep original qualified_field when wrap_projections

* only mutate the table reference when sending plan to execute

* formatting

* rename method
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants