Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize the TableVectorizer column-wise #592

Merged
merged 62 commits into from
Sep 20, 2023
Merged
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
eb0a020
fall back to pandas if no datetime format is found
LeoGrin Jun 10, 2023
f813fbb
change changelog
LeoGrin Jun 10, 2023
5d5e862
Merge branch 'main' of https://github.com/skrub-data/skrub into table…
LeoGrin Jun 12, 2023
d9a374f
first working version
LeoGrin Jun 12, 2023
ffa9755
add tests
LeoGrin Jun 12, 2023
5bcb6dd
update changelog
LeoGrin Jun 12, 2023
c005a49
typo
LeoGrin Jun 12, 2023
85b7f9c
copy transformer when split between columns to avoid conflict
LeoGrin Jun 13, 2023
26e66bd
Merge branch 'main' into table_vec_parallelism
LeoGrin Jun 13, 2023
c3706c7
update changelog
LeoGrin Jun 12, 2023
4147549
fix bug with repeated transformer
LeoGrin Jun 13, 2023
7771eb6
split and merge
LeoGrin Jun 22, 2023
8acca83
more tests
LeoGrin Jun 22, 2023
8dd62cb
merge with main
LeoGrin Jun 22, 2023
b997d28
fix test
LeoGrin Jun 23, 2023
f1ec2dd
also split self.transformers_ to parallelize transform
LeoGrin Jun 23, 2023
198fb04
merging with latest change
LeoGrin Jun 26, 2023
4780dfc
add _split to minhashencoder
LeoGrin Jun 26, 2023
744e881
cleaning
LeoGrin Jun 27, 2023
856892b
clean test
LeoGrin Jun 27, 2023
4cb44a5
merge with main
LeoGrin Jul 17, 2023
e50323a
fix wrong merge
LeoGrin Jul 17, 2023
6d007cf
fix wrong merge
LeoGrin Jul 17, 2023
f2b4266
Merge branch 'main' into table_vec_parallelism
LeoGrin Jul 18, 2023
970a73a
Apply suggestions from code review
LeoGrin Jul 19, 2023
d15e7e5
Update skrub/_utils.py
LeoGrin Jul 19, 2023
4ae9ae2
apply Lilian's suggestions
LeoGrin Jul 19, 2023
b5848e7
add future annotation to avoid circular import in type hints
LeoGrin Jul 19, 2023
6f645fc
add docstring
LeoGrin Jul 19, 2023
9753bd0
Update skrub/_table_vectorizer.py
LeoGrin Jul 20, 2023
6e48fab
type hint
LeoGrin Jul 20, 2023
76ea8a9
Merge branch 'table_vec_parallelism' of https://github.com/LeoGrin/sk…
LeoGrin Jul 20, 2023
63e5afd
revert change
LeoGrin Jul 20, 2023
3b768ea
Merge branch 'main' into table_vec_parallelism
LeoGrin Jul 20, 2023
20682dd
Merge remote-tracking branch 'upstream/main' into table_vec_parallelism
LeoGrin Jul 27, 2023
2fc38b9
Merge branch 'table_vec_parallelism' of https://github.com/LeoGrin/sk…
LeoGrin Jul 27, 2023
5e6966a
first batch of Vincent's suggestions
LeoGrin Aug 4, 2023
da06783
use tags
LeoGrin Aug 4, 2023
3671fb6
Merge branch 'main' into table_vec_parallelism
LeoGrin Aug 4, 2023
f8b177a
compare fitted transformers better
LeoGrin Aug 4, 2023
e1a98ea
Merge branch 'table_vec_parallelism' of https://github.com/LeoGrin/sk…
LeoGrin Aug 4, 2023
02bf26e
talk about tags in changelog
LeoGrin Aug 4, 2023
bb7ad81
run precommit checks
LeoGrin Aug 4, 2023
9dd62bb
get rid of _transformers_original
LeoGrin Aug 8, 2023
d6939f5
clean tests
LeoGrin Aug 18, 2023
9157451
Merge remote-tracking branch 'upstream/main' into table_vec_parallelism
LeoGrin Aug 25, 2023
99b3a9b
Apply suggestions from code review
LeoGrin Aug 26, 2023
769df3a
_parallel_on_columns
LeoGrin Aug 28, 2023
28cfb4c
explain transformers vs transformers_
LeoGrin Aug 29, 2023
e338fe1
Merge branch 'table_vec_parallelism' of https://github.com/LeoGrin/sk…
LeoGrin Aug 29, 2023
9aec058
split merge into two functions
LeoGrin Aug 29, 2023
72dd3d4
add tests to check that splitting doesn't prevent resetting transformers
LeoGrin Aug 29, 2023
2d5fd3a
Apply suggestions from code review
LeoGrin Aug 30, 2023
2eb6f5b
combine merge_unfitted and merge_fitted and move _split outside of class
LeoGrin Aug 30, 2023
7f7b524
don't return empty new_transformer_to_input_indices
LeoGrin Aug 30, 2023
ebcca20
remove future warning
LeoGrin Aug 30, 2023
d8e10ce
remove future warning
LeoGrin Sep 1, 2023
25d854b
add docstrings
LeoGrin Sep 1, 2023
0d381d0
fix merge
LeoGrin Sep 1, 2023
1cef43d
fix test
LeoGrin Sep 1, 2023
eabdb8b
Update skrub/_table_vectorizer.py
LeoGrin Sep 1, 2023
134a413
fix type hints
LeoGrin Sep 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion skrub/_table_vectorizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
manually categorize them beforehand, or construct complex Pipelines.
"""

import copy
import warnings
from itertools import chain
from typing import Dict, List, Literal, Optional, Tuple, Union
Expand Down Expand Up @@ -495,7 +496,7 @@ def replace_passthrough(name, trans, columns):
# selection is done with one dimension
cols = cols[0]

yield (name, trans, cols, get_weight(name))
yield (name, copy.deepcopy(trans), cols, get_weight(name))
LeoGrin marked this conversation as resolved.
Show resolved Hide resolved

def _more_tags(self):
"""
Expand Down