[help] How to stop an outdated target from invalidating its downstream target?

[Jiefei-Wang]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I have a list of connections that are used for collecting data. However, the number of connections will grow as the project grows, so what I'm trying to build is a rule such that

1. If the downstream target of the connections is successful, then never rerun it even if the connections itself is outdated
2. If the downstream target fails, then rerun whenever tar_make()

Because of this, I try to move my connections out of tar_target. My code is something like this

cl <- list(
    con1 = 1, ## just a placeholder, it is actually a connection
    con2 = 2  ## there will be more than 100+ connections in practice
)

foo1 <- function(cl){} ## do something using connections
foo2 <- function(cl){} ## otherthings
foo3 <- function(cl){} ## otherthings

list(
  tar_target(table1, foo1(cl)),
  tar_target(table2, foo2(cl)),
  tar_target(table3, foo3(cl))
)

This will result in a graph like this

As you can see, even if I put cl outside the target list, it is still being tracked. Therefore, whenever I update cl, tar_make will rerun every target even though I know they will never use the new connections. I see there is a cue function that can stop the effect of upstream changes, but is there anything I can do to stop cl from affecting its downstream? Thank you!

[wlandau]

To make a target not rerun when its upstream dependencies change, you could supply cue = tar_cue(depend = FALSE) to tar_target().

But in general, file/database connection objects are not exportable. This means they do not easily ship to parallel workers. I would recommend creating each connection within the target itself, i.e.:

run_target <- function() {
  con <- create_connection()
  out <- do_work(con)
  close(con)
  return(out)
}

with:

tar_target(your_target, run_target())

[Jiefei-Wang]

Thanks for your reply!

If I understand the documentation correctly, I think there might be a feature request: If I supply cue = tar_cue(depend = FALSE) to tar_target(), it will stop a target from updating itself forever unless I change its code. Instead of simply giving a logical value to depend, is it possible to supply a character vector where each element is the name of the upstream objects? So a user can specify which upstream object should not affect the current target(in my case it will be cl). Thanks!

[wlandau]

Instead of simply giving a logical value to depend, is it possible to supply a character vector where each element is the name of the upstream objects?

This is not feasible for targets because each target's metadata keeps a single aggregated hash of all its dependencies, rather than tracking each hash of each dependency individually. This is important for keeping targets efficient.

[Jiefei-Wang]

I see, thanks for your explanation!