llama cannot handle repeated CVs

[berndbischl]

No description provided.

[berndbischl]

We could IN PRINCIPLE simulate repeated CVs with BatchExperiments.

Although this might be useless work, as handling this in llama out-of-box is still preferable, and "working around this" in this package might be ugly.

[larskotthoff]

I'm not sure if either this package or LLAMA should provide a facility for this. Providing CV folds as part of the task spec may be useful, but providing detailed specs for a whole series of experiments is going a bit too far I think.

[berndbischl]

If the data set is so small (or imbalanced or strange or whatever....) 10CV is not the best resampling splitting. So we have to select and store a better one on the server. And handle it in the experiments.

You were the one who suggested that he wants to look at the variance if we change the splitting?

[berndbischl]

The problem might be, that 10CV is OK, but we as scientists might worry and we would only stop worrying when we SEE that the variance is not so big.

Like I said, BatchExperiments would allow to do that in like 10 minutes of coding.

I will do this later (after submitting the paper), then we can study the effect.

[larskotthoff]

I completely agree, I just don't think that this should be part of the package itself.

[berndbischl]

I think we do this:
convert2llama will create a list of what it creates now, for the reps of the CV.
We will then iter over this, maybe directly with BE, like I said, then it is auto-parallel.