Extending ibis with higher level abstractions

[thegreymatter]

I've bee working with ibis in the last few weeks for several workflows and was very impressed. as part of my work i've developed several "common" utility functions that leverage ibis base expressions.

just few examples -
pivot, one hot encoding (i.e pandas get_dummies), winsorizing , merge overlaps in time series, gaps&islands in time series e.t.c.

The common theme here is that all of these expressions are syntactic-sugar of a more complex expression that can be written manually as ibis expressions. (even though some backends have direct implementation for them)

I would be happy to hear your thoughts about the how such capabilities play with the core framework.

1. Does the core framework has a place for common higher level abstractions like these.
2. If not, and we are looking into an entire ecosystem of extension packages. What is the recommended approach for integrating them into the core framework while keeping the UX of "Everything can be method chained"

[cpcloud]

Hey @thegreymatter! Great to hear that you're enjoying the library. We've been going gangbusters in a few different areas so it's very gratifying to see that you were able to get things done and also were able to build some extensions.

I'll just jump right in:

Does the core framework has a place for common higher level abstractions like these.

Not specifically, but if there's a natural ibis.expr.types.Value subclass that they can be attached to or if they make sense on a TableExpr then that would be the place to define them.

If not, and we are looking into an entire ecosystem of extension packages. What is the recommended approach for integrating them into the core framework while keeping the UX of "Everything can be method chained"

One lightweight approach might be to leverage the Expr.pipe method and write the extensions as functions:

t = ...
t.pipe(onehot).pipe(...).select(x=_.a.winsorize())

This allows you to keep the method chaining UX (mostly!) while not having to add a ton of methods. In that case, we could potentially define an ibis.ext inside the main repo where we could keep these utilities.

[cpcloud]

FWIW, I think the operations that can be done on columns should definitely be methods built into the core!

[cpcloud]

@thegreymatter What do you think about PR-ing the simplest extension you have, and then we can go from there?

[cpcloud]

@thegreymatter Any updates here? Would love to see what you've done!

[thegreymatter]

Sorry for the slight delay.

Sharing some of the "extensions" we were writing and using extensively in the last few weeks. not all of them (actually most of them) are not worthy as a core capability.
https://github.com/thegreymatter/ibis-extensions/

yet, when looking on how other libraries handle this issue, i've encountered this interesting approach implemented by polars. which is much friendlier that "pipe" approach.
https://pola-rs.github.io/polars/py-polars/html/reference/api.html

will be happy for your thoughts, and since its a relatively straight forward impl, i'll be more than happy to PR it if it makes sense.

[IndexSeek]

I know this discussion has been quiet for a while; I was looking into what it would take to make get_dummies work in Ibis. I like the implementation of the OneHotEncoder class; it feels like scikit-learn.

Could we try with an implementation of get_dummies (to_dummies)? I see you have one defined that is using that aforementioned class.

https://github.com/thegreymatter/ibis-extensions/blob/bc83115ca5ee33e6707dc078163f2f9667c5223b/ibis_extensions/dummies.py#L48C8-L71

I put one together awhile back that follows this sort of pattern:

import ibis
from ibis import _
from ibis import selectors as s

t = ibis.read_csv("titanic.csv")

embarked_categories = t.select("Embarked").distinct().to_pandas().Embarked.to_list()[1:]

t.mutate(
    s.across(
        s.c("Embarked"),
        {
            f"{x}": _.case().when(x, 1).else_(0).end()
            for x in embarked_categories
        },
    ),
).drop("Embarked")

The challenge is maintaining similarity between the existing APIs. Pandas can behave slightly differently from scikit-learn, especially in the defaults and preferences and optional settings.

An example would be how I was slicing off a column to emulate OneHotEncoder; the categories parameter specifies the unique values for each category (in this case, the unique values of “Embarked”), and the drop parameter is set to “first” to drop the first category in each feature. This avoids multicollinearity, but I think Pandas brings all the columns along.

I would be happy to try this out with a PR if we consider it worthwhile or happy to contribute to the existing work.

Edit: for reference, how I ended up here in train of thought: https://docs.rs/polars/latest/polars/prelude/trait.DataFrameOps.html#method.to_dummies

[NickCrews]

This is an important discussion, thanks for starting it!

Here are my opinions, not sure how unpopular these are. Please comment if you agree/disagree!

Direct methods (eg Table.to_onehot(), Table.merge_timeframes())

I think this is great for very common operations (eg pivot() and dedup(). These are now core methods in ibis, which I think is great!), but I don't like obscure or domain-specific ones polluting the namespace (eg merge_timeframes(), to_onehot()). Maybe the guiding principle is the function needs to be used by > 5% of people/usecases for it to be included? IDK exactly.

Namespaced methods (eg Table.time.merge_timeframes(), Table.ml.to_onehot())

I don't like these magic accesses, and I don't think they should ever be used.

Is time a column? or a namespace? If a 3rd party extension registers it in some distant .py file, then I need to know the execution environment in order to tell the difference. I need to look up where the extension was registered, and then track down the name of the python package, and then I can Google the API of that extension. Perhaps this could be mitigated by suggesting/requiring that packages on pypi match, eg ibis-time provides the time namespace, ibis-ml provides the ml namespace.

These also don't play well with typing, autocomplete, IDEs, etc

This could be acceptable if the namespace is builtin, eg if ibis ships with time and ml builtin then every user knows where to lookup the API. But still it risks collisions with common column names like time. As an example, there was some past issue here where the function .count() was getting shadowed by a column "count" coming from .topk(). Better might be just underscored prefixes like Table.ml_onehot() or Table.time_merge(), then they will all be grouped together in the API docs.

Direct functions (eg merge_timeframes(Table), onehot(Table))

This doesn't allow for the method chaining, but I think the directness and unambiguity is worth it for any 3rd party extensions. Using .pipe() is fine IMO, maybe advertise it's availability a bit better.