Allow OptunaSearchCV without Cross Validation (_n_splits = 1)

[sgerloff]

Motivation

The current implementation is incompatible with sklearn.model_selection.PredefinedSplit, which allows for having a single train-test split of the data. (e.g. optuna/optuna#5196) This splitting strategy can become necessary when dealing with domain shift (for example).

Description

I propose to wrap the call to report_cross_validation_scores in a try-except block, catching the ValueError. In case _n_splits = 1, a UserWarning should be raised, explaining the problem to the user, but continue anyway. If _n_splits > 1 the caught error is raised again as it does seem to originate from another issue. This way the user can decide to use OptunaSearchCV for CV's with a single split despite the potential incompatibility with optional callbacks.

The warning message could contain information on:

• incompatibility with TerminatorCallback which relies on CrossValidationErrorEvaluator
• general incompatibility with callbacks expecting _CROSS_VALIDATION_SCORES_KEY in the Trial.system_attr
• the ability to silence the warning with warnings.filterwarnings

Alternatives (optional)

• parameterize the _Objective, to enable users to provide a compatible _Objective for single split CV's
• parameterize a boolean flag that turns off the call to report_cross_validation_scores (this could be coupled with additional checks on the callbacks which raise warnings if both the call to report_cross_validation_scores is turned off and a TerminatorCallback is provided.
• allow users to customize the behavior of _Objective.fit in other ways

Additional context (optional)

No response