Pandas as an optional dependency

[ogrisel]

Now that Ibis uses duckdb as its default backend, it is possible to no longer import pandas by default (see #4786).

As a consequence @cpcloud suggested making pandas an optional dependency:

• perf(ux): remove pandas import overhead from import ibis #4786 (comment)

But then, what kind of datastructure would be returned by .execute() by default? Shall the type of the datastructure be configurable (e.g. choose between pandas.DataFrame or pyarrow.Table for instance)?

[ogrisel]

Personal opinions:

• having a return type that depends on the backend would be potentially very troublesome.
• pandas.DataFrame are very well integrated in the Python ecosystem e.g.:
  • nice displays in jupyter notebooks, VS Code...
  • good support in plotting libraries and includes matplotlib integration in the .plot attribute.
• pandas also supports various serialization format which makes it particularly easy to export the results of an ibis query to anythink with very little extra code.

Therefore switching to pyarrow.Table might negatively impact the usability of ibis quite significantly.

[cpcloud]

I was thinking that to ensure compatibility a DataFrame would be the default, but just like the default backend is configurable the default return type would be configurable.

It's possible that we can't make pandas truly optional, and the best we can do is reduce its surface area inside ibis and make everything go through pyarrow (followed by a call to to_pandas).

Agree that DataFrames have a strong ecosystem pull, and losing that by default would likely be detrimental to usability.

[ogrisel]

From a packaging point of view it could makes sense to package an ibis-framework-core (both on pypi and conda-forge) package without the pandas dependency for people who use ibis for non-interactive ETL workfloads where pyarrow is enough while keeping the default ibis-framework package on pypi and conda-forge would still come with the pandas dependency for usability concerns.

[cpcloud]

Thanks for the discussion @ogrisel! I think we're going to leave things as they are right now. We have a decent amount of work to do before we could consider making pandas an optional dependency (and it's not clear we should anyways!).