Branching over static branches in large pipeline, tar_combine() requires existing objects

[adamaltmejd]

Migrating a large pipeline over from Drake where I was doing a lot of static branching in different steps, loading files and then later combining them in different ways. With static branching in targets I have to first save the branches as a targets object that i then supply to tar_combine(). This disrupts the readability of my code a lot since i need to create lots of objects with different names. In targets, it seems its a lot simpler to do this with dynamic branching - the only downside being that the targets get names that are impossible to interpret. Is there any way to give dynamic targets custom names?

Here are a couple of examples to illustrate the problem, of course, my real use cases are a lot more complicated:

This thus not work, but is what I started trying out when coming from drake:

targets::tar_dir({
targets::tar_script({
    list(
      tar_target(df, data.frame(x = 1:100, y = rnorm(100), gr = sample(1:3, 100, replace = TRUE))),
      tarchetypes::tar_map(
        values = list(group = c(1, 3)),
        targets::tar_target(filtered_df, dplyr::filter(df, gr == group))
    ),
    tarchetypes::tar_combine(combined_df, filtered_df, command = dplyr::mutate(rbind(!!!.x), y2 = y^2))
    )
})
targets::tar_manifest()
})

It of course throws object 'filtered_df' not found. So I tried putting the tar_map() code inside tar_combine() which does not work as tar_combine() only returns the combined target:

targets::tar_dir({
targets::tar_script({
    list(
        tar_target(df, data.frame(x = 1:100, y = rnorm(100), gr = sample(1:3, 100, replace = TRUE))),
        tarchetypes::tar_combine(
            combined_df,
            tarchetypes::tar_map(
                values = list(group = c(1, 3)),
                targets::tar_target(filtered_df, dplyr::filter(df, gr == group))
            ),
        command = dplyr::mutate(rbind(!!!.x), y2 = y^2))
    )
})
targets::tar_make()
})

When one attempts a build it throws Error : object 'filtered_df_1' not found. To get it to work I need to assign parts of the pipeline to objects first:

targets::tar_dir({
targets::tar_script({
    target1 <- tar_target(df, data.frame(x = 1:100, y = rnorm(100), gr = sample(1:3, 100, replace = TRUE)))
    mapped <- tarchetypes::tar_map(
        values = list(group = c(1, 3)),
        targets::tar_target(filtered_df, dplyr::filter(df, gr == group))
    )
    combined <- tarchetypes::tar_combine(combined_df, mapped, command = dplyr::mutate(rbind(!!!.x), y2 = y^2))
    list(target1, mapped, combined)
})
targets::tar_manifest()
#targets::tar_make()
#targets::tar_read(combined_df)
})

But this is a lot messier. I need to keep track of the objects, mapped and combined and if I want to keep the pipeline list() in _targets.R organized I need to split parts of the code that are not even being mapped too. So, I've now changed most of my code using such branching in Drake to use the following style with dynamic branching:

targets::tar_dir({
targets::tar_script({
    list(
        tar_target(df, data.frame(x = 1:100, y = rnorm(100), gr = sample(1:3, 100, replace = TRUE))),
        tar_target(group, c(1, 3)),
        tar_target(filtered_df, dplyr::filter(df, gr == group), pattern = map(group)),
        tar_target(combined_df, dplyr::mutate(filtered_df, y2 = y^2))
    )
})
#targets::tar_manifest()
targets::tar_make()
targets::tar_read(combined_df)
})

The only downsides being that I need to create an extra target for "group" and of course that the branch names are uninformative. What would you say is the best way to run these kinds of workflows? Is there any way to assign names to dynamic targets?

[wlandau]

With static branching in targets I have to first save the branches as a targets object that i then supply to tar_combine(). This disrupts the readability of my code a lot since i need to create lots of objects with different names.

It may be an adjustment, but this kind of static branching is far more reliable than what drake had. When you do use static branching, I still recommend that you define intermediate objects as the examples demonstrate.

Is there any way to give dynamic targets custom names?

No, because we cannot always parse the names of the objects we branch over. And even if we could, dynamic branching assumes we do not know the branches in advance, so individual branch names should not be used to construct the pipeline to begin with.

In practice, static branching operates best at a high level when there are few things to branch over, and dynamic branching is better suited to handle larger numbers of branches. That last example you wrote looks nice. If you find yourself wanting the names of dynamic branches, I would recommend instead saving any identifying metadata you need in the branch itself instead of encoding it in the name. If you want to select groups of individual branches programmatically, pattern = slice(...) can help if you know the branch indexes in advance.

[adamaltmejd]

Thanks, very helpful!