Cross-validation not seeded implies wrong Prediction Intervals

[vincentblot28]

Hello, I encountered a problem in MAPIE. When we use MapieRegressor with J+ or CV+ without any random state, then the Kfold and LeaveOneOut methods of sklearn are not seeded. It means that when computing the $R_i^{LOO}$, it happens that we compute them with which used the $i$th observation as training point. Here is a reproducible example to show that, without seed, when calling cv.split(X) at two different moments leads to different split of the data (which is the case in EnsembleRegressor)

from mapie.regression import MapieRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
X, y = make_regression(10)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2)

mapie = MapieRegressor(method="plus", cv=5)
mapie.fit(X_train, y_train)
for train_index, _ in mapie.estimator_.cv.split(X_train):
    print("Train index: ", train_index)
    
for _, cal_index in mapie.estimator_.cv.split(X_train):
    print("Cal index: ", cal_index)

[vincentblot28]

Here is a test that should pass once the bug is fixed:

def test_same_split_no_random_state():
    X, y = make_regression(10)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2)
    
    mapie = MapieRegressor(method="plus", cv=5)
    mapie.fit(X_train, y_train)
    train_indices_1 = []
    train_indices_2 = []
    for train_index, _ in mapie.estimator_.cv.split(X_train):
        train_indices_1.append(train_index)
    
    for train_index, _ in mapie.estimator_.cv.split(X_train):
        train_indices_2.append(train_index)
        
    for i in range(mapie.estimator_.cv.get_n_splits()):
        assert (train_indices_1[i] == train_indices_2[i]).all()

[thibaultcordier]

Thank you for reporting this bug. The problem should be solved directly in check_cv in utils.py as it is a wrapper around the cv attribute of the MAPIE estimator (called at fit time for both regression and classification). Instead, I suggest a test for this function.