invoking pipeline programmatically

[gofford]

tar_make() requires a list object at the end of the _targets.R file to invoke the pipeline. This is fine in the scenario that there is only one (self-contained) pipeline, or the user is happy to have all of their different values mapped within one very large pipeline.

Consider this example, where I have a complete pipeline factory called run_pipeline which accepts the arguments: label and config, where label is a string vector and config is a YAML file with various configuration parameters.

Currently, if run_pipeline() is located at the end of _targets.R I can invoke it with tar_make_*(). This is fine for one label. In reality, I have many instances of label across which the pipeline needs to be run. The obvious answer here is to pass all of the labels to tar_map() and iterate across them there, within one single pipeline. However, this isn't really viable because (a) there are many labels, which makes the DAG absolutely enormous, and (b) it means that all labels share the same metadata, and (c) having a single large DAG prevents the pipeline being run over multiple (unconnected) instances (i.e., that are not part of a cluster)

My desired solution here would be something like this:

iter_labels <- c("label1", "label2", "label3")
iter_labels %>%
    walk(function(label) {
        run_pipeline(label, config)
    })

This would invoke the pipeline 3 times. Normally this would clobber the metadata, but because the pipeline is set up such that each individual label is sync'd to S3 discretely, this isn't a concern.

Is this -- or something equivalent -- possible?

[wlandau]

I recommend other strategies such as batching, maybe multiple labels per target. Sketch with 10 reps within 10 batches:

# _targets.R
library(targets)
library(tarchetypes)
library(tibble)
list(
  tar_group_by(
    label_df,
    tibble(
      labels = sprintf("label%s", seq_len(100)),
      batch_id = rep(seq_len(10), each = 10)
    ),
    batch_id
  ),
  tar_target(
    per_batch,
    apply(label_df, 1, length), # Loop over the rows to do something for each label within batch.
    pattern = map(label_df) # One dynamic branch target per batch.
  )
)

As explained at https://books.ropensci.org/targets/practices.html#how-to-define-good-targets, there are tradeoffs in terms of how work is divided up, and it can slow things down to have a lot of targets that each do a small piece of the work.

Does that mean run_pipeline() invoke tar_make(), or does it just define a list of target objects? The latter is best, the former could cause trouble. If _targets.R ends with tar_make(), that causes infinite recursion because tar_make() itself sources _targets.R. There are safeguards in targets to try to prevent this, but they involve statically analyzing _targets.R for explicit mentions of tar_make() etc., so this problem could go unnoticed if run_pipeline() calls tar_make().

If you want to split up the work across multiple pipelines, they should have separate _targets.R files and separate _targets/ data stores. If multiple instances of tar_make() or multiple pipelines share the same _targets.R and _targets/, targets will break because I did not design it for this. Workarounds like batching can help define one pipeline for a large project. If even that is too much (not a common situation) then if the subject matter of your project naturally separates out the pipeline, then those pieces could be defined as different projects.

[wlandau]

That's surprising. Dependency detection uses codetools::findGlobals(), with the exception of special treatment for formulas to make purrr stuff work. Details are explained at https://books.ropensci.org/targets/practices.html#dependencies. The reprex following works for me. As you can see, there is a directed edge from target1 to target2. But target1 has to be in the pipeline to begin with in order for that relationship to be established.

library(targets)
tar_script({
  # _targets.R
  library(targets)
  library(tarchetypes)
  library(tibble)
  list(
    tar_target(target1, 1),
    tar_target_raw(name = "target2", command = quote(list(target1)))
  )
})

tar_visnetwork()

^{Created on 2021-08-05 by the reprex package (v2.0.0)}

[gofford]

So, this does work in the example you provided. But if the command variable of target2 has multiple elements wrapped in {...} (which works in a tar_target()) the dependency is not respected:

library(targets)
tar_script({
  # _targets.R
  library(targets)
  library(tarchetypes)
  library(tibble)
  list(
    tar_target_raw(
      name = "target1", 
      command = 1
    ),
    tar_target_raw(
      name = "target2", 
      command = {
        quote(list(target1))
        substitute(print(val), env = list(val = 'value'))
      }
    )
  )
})

tar_visnetwork()

It works if the substitute command isn't present, and there is only 1 element in the {...} With multiple args, only the last is reflected in the target$command$string. Any ideas?

[wlandau]

tar_target() quotes the command argument to create an expression. tar_target_raw() expects command to already be an expression. The code

{
  quote(list(target1))
  substitute(print(val), env = list(val = 'value'))
}

evaluates to an expression with

print("value")

so that target2 is equivalent to tar_target(target2, print("value")).

[wlandau]

So maybe you wanted something like this

substitute({
    list(target1)
    print(val)
  }, env = list(val = "value"))

which evaluates to an expression with

{
    list(target1)
    print("value")
}

[gofford]

💥 that's solved it nicely. Thanks @wlandau.