Equivalent of ColumnTransformer functionality remainder{‘drop’, ‘passthrough’} in feature-engine?

[noahjgreen295]

Hi,

One of the few things I miss about the sklearn feature pipeline is the ability for ColumnTransformer to dynamically drop all variables unused by the transformers it contains, via the remainder='drop' parameter setting. This allows you to throw arbitrary DataFrame instances at the pipeline and not have to explicitly drop the unused features, or worry about breaking things if you don't.

Is there an equivalent of this functionality in feature-engine? If not, could I put together a PR containing a proposed implementation?
(Sorry if it's there already and I just couldn't find it)

Thanks,
-Noah

[solegalli]

Hi @noahjgreen295

Is the idea that each transformer should drop the features that it does not use? or all unused features should be dropped at the end of the pipeline?

Could you provide an example of what is the intended functionality / result?

[noahjgreen295]

The idea would be that all unused features should be dropped at the end of the pipeline.

I put together a notebook showing it all, but looks like github won't allow me to attach it here.
Is there another way I could get it to you? Alt. I can paste all the code into reply

If you think this functionality is worth adding, I would be happy to work on something!

[noahjgreen295]

Ah, I was able to do it as a gist!

Updated it a few times. For some reason gist does not display all the output, so you'll still need to download and run it locally to see it all.

[solegalli]

Thank you for the details @noahjgreen295

I am trying to understand a bit more when we would need the requested functionality. Please help me with the following:

If you pass the variables to the different transformers, like in the sklearn pipe or in the feature engine pipe, you do have foreknowledge of which features you will use, in your example x1, x2 and x3. So why not passing dataframe[[x1,x2,x3]] to the pipeline instead of passing the entire dataframe and then dropping the variables at the end?

In the case of the feature-engine pipe, in fact, I would recommend passing DropFeatures at the beginning of the pipeline, so all the remaining transformers operate over smaller datasets, then it is more memory efficient.

You mention something about difficult in dynamic settings like deployment. I am having trouble trying to understand why you would want to avoid knowing which features to drop, when you would actually know which features to use/keep, and then you could just slice the dataframe to these features from the beginning?

Thank you

[noahjgreen295]

I apologize, deployment is a bad example, because at that point the model features would be finalized so it would be easy to know which features to maintain.

You're right, I should have thought also of memory efficiency, which would be compromised when dealing with a whole lot of unused data passing through the pipeline. Though one of the ideas I had for implementing this feature would actually automatically drop the unused at the beginning of the pipeline.

I guess I'm trying to avoid explicit feature selection outside the pipeline, so that the pipeline handles absolutely everything, including choosing the initial feature columns and filtering out those unused. This is possible only with ColumnTransformer. But in the end I guess it's really not such a big deal to have a step outside the pipeline that does that, so totally understood if you think this is not necessary.