How to superpose different misclassification cost term curves using cutpointr

[Thie1e]

I received the following question via e-mail and post the answer here for future reference:

I really like your package which is awesome to look for the accuracy of diagnostic markers.
I would like to look for the robustness of my marker findings relative with various situation of FN/FP cost ratio.
Do you know how can I overlay on the same figure the different results of plot_metric() function?

I want to overlay both curves on the same graph as a way to look for cutoff robustness across various FN/FP ratio.

An additional question I would have is on the possibility to look for the robustness of the approach using the same FN/FP ratio but applied to different prevalence setting. I don’t see it as a possibility when reading the package info but maybe I missed something?

Answer:

There's no built-in function for overlaying multiple metric curves for different metrics or costs, only for different subgroups. You can overlay them manually, though:

library(cutpointr)
library(tidyverse)

# Estimate two cutpoints with different costs -----------------------------

cp1 <- cutpointr(suicide, dsi, suicide, subgroup = gender,
                 metric = misclassification_cost,
                 cost_fp = 1, cost_fn = 1)
cp2 <- cutpointr(suicide, dsi, suicide, subgroup = gender,
                 metric = misclassification_cost,
                 cost_fp = 1, cost_fn = 5)

plot_metric(cp1)
plot_metric(cp2)

# The metric values are in roc_curve in column "m" -------------------

# We have to create a tibble that contains both results

dat <- bind_rows(
    cp1 %>%
        select(subgroup, roc_curve) %>%
        unnest(roc_curve) %>%
        mutate(model = "cost_fp = 1; cost_fn = 1"),
    cp2 %>%
        select(subgroup, roc_curve) %>%
        unnest(roc_curve) %>%
        mutate(model = "cost_fp = 1; cost_fn = 5")
)

ggplot(dat, aes(x = x.sorted, y = m, color = model)) +
    geom_point() +
    geom_line() +
    theme_bw() +
    xlab("Cutpoint") +
    ylab("Misclassification Cost") +
    facet_grid(~subgroup)

If you have a large number of models or data sets to compare, there are probably ways to use some form of map() to go over these models elegantly.

Regarding the prevalence, it's not a setting - that metric is simply calculated from the complete data. If you want to test your markers at different prevalences you would have to generate data sets with the appropriate prevalence percentages.