Serialization/De-Serialization of Expressions

[NickCrews]

Thank you for your help!

So I've been using Splink to do entity resolution. With Splink, you define join conditions using raw SQL strings (see blocking_rules_to_generate_predictions in the first few cells). They chose this method because it makes it trivial to save/load your model as JSON.

I would like to do something similar, but I want to

• define the join conditions using Ibis expressions, eg something like lambda left, right: left.first_name == right.last_name
• Be able to serialize this expression so I can save it to disk
• Be able to deserialize it. Ideally, be able to reconstruct the original expression, so it's possible to continue building/editing.

I see that it is possible to serialize an expression to substrait, but we can't round-trip that representation back to Ibis, at that point we can only pass that off to a SQL engine.

Compiling an expression to a raw SQL string would be great, but again there is no way to revert this back to an Ibis expression that I know of.

Any ideas?

[gforsyth]

Hey @NickCrews --
ibis-substrait also has a decompiler that goes from substrait to ibis expressions -- that said, it's out-of-date and I'm not sure we're going to keep it around (although if that makes the most sense for deserializing query plans we'll need to reconsider).

As for raw SQL, check out #4853
There's support for parsing SQL into an ibis expression, and then further experimental support for generating the Python code that would produce that expression.

[NickCrews]

You people have thought of everything! Thanks, I'll check those two options out!