measures in LOO

[jakob-r]

For leave-one-out resampling it might for some measures not so feasible to calculate the measure on each point and then aggregate it.

Let's for example look at LOO: Here the AUC can not be calculated for this example:

task = sonar.task
lrn = makeLearner("classif.ranger", predict.type = "prob")
rdesc = makeResampleDesc("LOO")
res = resample(learner = lrn, task = task, resampling = rdesc, measures = auc)
res

But IMO it makes sense to see all predictions as one set and calculate the AUC over the complete set of all predictions.
I am sure that they are even cases in the literature where the same principle is applied for CV.

[berndbischl]

1. first of all, this is supported, simply use aggregation function "test.join" (and read its documentation now pls)

2. the remaining issue why this still not works for auc (based on probabilities) is this here:
   test.join aggregation does not join probabilities #625

so i will close his now.
reopen if you think docs are "broken" and help to improve them

and note that issue 625 is marked as "prio severe"