Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python-package] fix retrain on sequence dataset #6414

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[python-package] fix retrain on sequence dataset #6414

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2b7811b
fix retrain on sequence dataset
eromoe Apr 11, 2024
a07800c
add testing for incremental training on Dataset with lgb.Sequence
eromoe Jul 14, 2024
3ac186c
Update python-package/lightgbm/basic.py
eromoe Jul 15, 2024
ecd5746
Update tests/python_package_test/test_sequence.py
eromoe Jul 15, 2024
b3bcf37
move seqence test to test_basic
eromoe Jul 15, 2024
48f062c
add test for seq_ds
eromoe Jul 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions python-package/lightgbm/basic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1192,10 +1192,16 @@ def predict(
predict_type=predict_type,
)
elif isinstance(data, list):
try:
data = np.array(data)
except BaseException as err:
raise ValueError("Cannot convert data list to numpy array.") from err
if isinstance(data[0], Sequence):
try:
data = np.concatenate([i[:] for i in data])
except BaseException as err:
raise ValueError('Cannot convert Sequence list to numpy array.') from err
else:
try:
data = np.array(data)
except BaseException as err:
raise ValueError('Cannot convert data list to numpy array.') from err
preds, nrow = self.__pred_for_np2d(
mat=data,
start_iteration=start_iteration,
Expand Down
37 changes: 37 additions & 0 deletions tests/python_package_test/test_basic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,43 @@ def test_sequence_get_data(num_seq):
np.testing.assert_array_equal(subset_data.get_data(), X[sorted(used_indices)])


def test_retrain_list_of_sequence():
X, y = load_breast_cancer(return_X_y=True)
seqs = _create_sequence_from_ndarray(X, 2, 100)

seq_ds = lgb.Dataset(seqs, label=y, free_raw_data=False)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was free_raw_data=False necessary here? If it wasn't, please remove it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If free_raw_data=True, model2 cannot get the data, would raise Exception I remember .


assert sum([len(s) for s in seq_ds.get_data()]) == X.shape[0]
assert len(seq_ds.get_feature_name()) == X.shape[1]
assert seq_ds.get_data() == seqs
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These checks should be moved after training, to avoid this test failure: https://github.com/microsoft/LightGBM/actions/runs/9935170010/job/27451324230?pr=6414

FAILED tests/python_package_test/test_basic.py::test_retrain_list_of_sequence - Exception: Cannot get data before construct Dataset

Please run the tests yourself before pushing another commit.

sh build-python.sh bdist_wheel install
pytest tests/python_package_tests/test_basic.py::test_retrain_list_of_sequence

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I only tested the code on jupyter notebook for this case.
Sorry I don't have env for sh , so can not build whell to run the pytest.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I will push testing changes for you.


params = {
"objective": "binary",
"num_boost_round": 20,
"min_data": 10,
"num_leaves": 10,
"verbose": -1,
}

model1 = lgb.train(
params,
seq_ds,
keep_training_booster=True,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
keep_training_booster=True,

Using keep_training_booster=True here works if the initial model (to be passed to init_model) is going to be passed onto training later in memory in the same process, but is that the situation that led to #6413.

I expect it will be more common to instead want to continue training with a model loaded from a file + a Sequence object in memory.

Could you please modify this test to not use keep_training_booster=True, or explain why it's necessary?

Copy link
Author

@eromoe eromoe Jul 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because I have a rolling timeseries trainning project.
The dataset is too large to load in memory. Aim to use memory efficiently, I read 12 month data and build model , make prediction for one month in future, then update datasource(scolling one month) and retrain model.
Retrain model only use recent 12 months data, means old model with old weight would only be updated by recent data.

model = None
for idx, (train_idx, test_idx) in enumerate(scroll_train_test(dates_partition, train_size=TRAIN_LOAD_STEP, test_size=TEST_LOAD_STEP, align_idx=train_end_idx)):
    
    train_partitions = dates_partition[train_idx]
    test_partitions = dates_partition[test_idx]

    train_df = read_partitioned_df(train_partitions, pre_train_partitions, train_df)
    test_df = read_partitioned_df(test_partitions, pre_train_partitions, test_df)
    ....
        model = lgb.train(
            params,
            train_data,
            init_model=model,
            num_boost_round=num_boost_round,
            keep_training_booster=True,
        )

Since it is in the loop, no necessary to dump model as a file, I just reuse it .

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for explaining that. Very interesting use of Sequence!

But the fact that you want to use this functionality in one specific way (with the model held in memory the entire time) does not mean that that's the only pattern that should be tested.

It's very common to use LightGBM's training continuation functionality starting from a model file... for example, to update an existing model once a month based on newly-arrived data. It's important that all LightGBM training-continuation codepaths support that pattern.

Anyway, like I mentioned in #6414 (comment), I can push testing changes here. Once you see the diff of the changes I push, I'd be happy to answer any questions you have.

)

assert model1.current_iteration() == 20
assert model1.num_trees() == 20

model2 = lgb.train(
params,
seq_ds,
init_model=model1,
)

assert model2.current_iteration() == 20
assert model2.num_trees() == 20
Comment on lines +253 to +254
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
assert model2.current_iteration() == 20
assert model2.num_trees() == 20
assert model2.current_iteration() == 40
assert model2.num_trees() == 40

These don't look correct. Performing training once with "num_boost_round": 20, then continued training again with "num_boost_round": 20, should result in a model with 40 boosting rounds.



def test_chunked_dataset():
X_train, X_test, y_train, y_test = train_test_split(
*load_breast_cancer(return_X_y=True), test_size=0.1, random_state=2
Expand Down
Loading