Understanding tar_cancel with dynamic branching

[robitalec]

Hello,

Looking to selectively cancel targets based on a condition, eg. in this case if a certain column has the character 'A'.

The _targets.R based on the example.

library(targets)

summ <- function(dataset) {
  tar_cancel('A' %in% dataset$z)
  summarize(dataset, mean_x = mean(x))
}

# Set target-specific options such as packages.
tar_option_set(packages = 'dplyr')

# End this file with a list of target objects.
list(
  tar_target(data, data.frame(x = sample.int(100), y = sample.int(100), z = LETTERS[1:5])),
  tar_target(split, data %>% group_by(z) %>% tar_group(), iteration = 'group'),
  tar_target(summary, summ(split), map(split))
)

Output from tar_make()

R: targets::tar_make()
• start target data
• built target data
• start target split
• built target split
• start branch summary_66865b4f
• cancel branch summary_66865b4f
• start branch summary_5d669193
• built branch summary_5d669193
• start branch summary_a2141425
• built branch summary_a2141425
• start branch summary_fbe2f44d
• built branch summary_fbe2f44d
• start branch summary_a7593665
• built branch summary_a7593665
• end pipeline

Note we don't get the message "built pattern summary" and tar_read can't read the full summary object. It can however read individual branches.

tar_read(summary)
Error: branches not in metadata: summary_388c2575

tar_read(summary, 3)
# A tibble: 1 x 1
  mean_x
   <dbl>
1   43.6

I'm trying to selectively cancel dynamic branches if they meet a certain criteria.
Then hoping to also pass those dynamic branches that aren't cancelled on to other targets (dynamic again).

Thanks!

[wlandau]

In your case, instead of tar_cancel(), I recommend returning a NULL value early.

summ <- function(dataset) {
  if("A" %in% dataset$z) return()
  summarize(dataset, mean_x = mean(x))
}

Then, the code in downstream dynamic targets could be written to handle NULL branches.

tar_cancel() makes no attempt to ensure that a target's value exists, and it makes no attempt to modify downstream dynamic branching behavior. I still like this limited scope because it is not likely to surprise users (surprises more often come from trying to do too much than too little) or create reproducibility problems far downstream. It also avoids a serious redesign on my end for the sake of a feature that would not come up very often.

tar_cancel() is really for niche situations: for example, when a target is trying to download data from an unreliable API endpoint and you want some way to fall back on the previous run's data if the current API call fails.

[wlandau]

Better yet, you could filter the upstream data in advance so as to only branch over the stuff you know will work. In fact, this capability is one of the primary goals of dynamic branching.

[robitalec]

Thank you for the information Will.
The intention or logic behind tar_cancel makes more sense to me now. And you're right, that doesn't line up with what I'm looking to do.

Ok, a NULL return could definitely work. And good point, filtering before the dynamic branching would be the ideal case. My actual targets workflow has a couple of long running steps, which (for some dynamic branches) return results which tell us these branches should be skipped. Say, for example, we prepare and filter data, then set up a dynamic branch to model distributions of dataframes, and only continue on with dataframes that fit some distribution.

1. Read and prepare data
2. Branch data based on some grouping
3. Fit a model to determine distributions
4. If distributions match expectations, continue to further analysis. If not, skip this dynamic branch
5. Downstream analyses, some might have further dropping of more dynamic branches

[robitalec]

I adapted your recommendation to check for NULLs and added a couple more steps.
I think checking for NULLs works, but adds what feels like a bit of redundant code, necessary for all downstream steps.

library(targets)
library(tarchetypes)

summ_x <- function(dataset) {
	if("A" %in% dataset$z) return()
	summarize(dataset, mean_x = mean(x))
}

plot_x <- function(dataset) {
	if("A" %in% dataset$z) return()
	qplot(x, y, main = dataset$z, data = dataset)
}

render_with_plot <- function(plot, out) {
	if(is.null(plot)) return()
	rmarkdown::render('report.Rmd', output_file = out, params = list(theplot = plot))
}

# Set target-specific options such as packages.
tar_option_set(format = 'qs', packages = c('dplyr', 'ggplot2'))

# End this file with a list of target objects.
list(
  tar_target(data, data.frame(x = sample.int(100), y = sample.int(100), z = LETTERS[1:5])),
  tar_target(split, data %>% group_by(z) %>% tar_group(), iteration = 'group'),
  tar_target(summary, summ_x(split), map(split)),
  tar_target(plots, plot_x(split), map(split), iteration = 'list'),
  tar_target(renders, render_with_plot(plots, tar_name()), map(plots))
)

[robitalec]

An alternative is doing the check for NULLs once, after a step that generates them.
Then I can subset on the combined results, and move on without checking at each downstream target if there's a NULL provided.

library(targets)
library(tarchetypes)

summ_x <- function(dataset) {
  mnx <- summarize(dataset, mean_x = mean(x), id = unique(z))
  if (mnx$mean_x < 44) return()
  mnx
}

check_nulls <- function(x, dataset) {
  chk <- !vapply(x, function(x) is.null(x$mean_x), TRUE)
  chkgroups <- seq.int(length(chk))[chk]
  dataset %>% filter(tar_group %in% chkgroups) %>%
    group_by(z) %>% tar_group()
}

plot_x <- function(dataset) {
  qplot(x, y, main = dataset$z, data = dataset)
}

# Set target-specific options such as packages.
tar_option_set(format = 'qs',
               packages = c('dplyr', 'ggplot2', 'rmarkdown'))

# End this file with a list of target objects.
list(
  tar_target(data, data.frame(
    x = sample.int(100),
    y = sample.int(100),
    z = LETTERS[1:5]
  )),
  tar_target(filtered, data %>% filter(z != 'A')),
  tar_target(split, filtered %>% group_by(z) %>% tar_group(), iteration = 'group'),
  tar_target(summary, summ_x(split), map(split), iteration = 'list'),
  tar_target(checksumm, check_nulls(summary, split), iteration = 'group'),
  tar_target(
    plots,
    plot_x(checksumm),
    map(checksumm),
    iteration = 'list'
  ),
  tar_target(
    renders,
    render(
      'report.Rmd',
      output_file = tar_name(),
      params = list(theplot = plots)
    ),
    map(plots)
  )
)

Part of this has to do with vctrs::vec_c dropping NULLs when the dynamic branches are aggregated.
So an is.null check on the aggregated dynamic branch doesn't actually do anything.
I don't love my solution here with a convoluted check if null, then re-doing the tar_group.
I think an is.null check in downstream targets like above is more obvious and explicit.