does tar_make_future support dynamic branching?

[kkmann]

Hi,

I am a bit puzzled since

library(targets)
library(future)

plan(multisession)
tar_make_future(workers = 12L)

does not seem to run my dynamic branching target in parallel (cpu load) - is that by design or did I miss something?
Here is my _targets.R

library(targets)

list(
	tar_target(genes, 1:12),
	tar_target(outcomes, c("y", "z")),
	tar_target(test,
		{
			rnorm(1e8)
			sprintf("%i_%s", genes, outcomes)
		},
		pattern = cross(genes, outcomes)
	)
)

How do I run a dynamic branching target in parallel? This is not explicitly documented in the docs...

[kkmann]

Ok, it seems that it is crucial to load future in the _target.R script. I must have missed that in the docs and it seems a very weird approach since future usually encourages seperation of backend definition definition and code. If I haven't just missed that, it would be extremely helpful to add that to the docs, seems important!

[wlandau]

Right, tar_make_future() requires future::plan(...) in _targets.R in order to be effective. Existing mentions in the docs:

1. The top of the tar_make_future() help file: "This function is like tar_make() except that targets run in parallel with transient future workers. It requires that you declare your future::plan() inside the _targets.R script."
2. The examples of tar_make_future() demonstrate the use of a future::plan().
3. Sections https://books.ropensci.org/targets/hpc.html#future-locally and https://books.ropensci.org/targets/hpc.html#future-remotely in the user manual.

[kkmann]

Apologies, shouldn't have missed it in the help. Amazing package btw - feels much more mature than drake. Thanks a lot for sharing your hard work!

It might still be helpful to add that to https://books.ropensci.org/targets/hpc.html#future-locally as well. I guess I don't yet fully understand how the _target.R script interacts with tar_make(). _target.R seemed to be almost declarative to me, in which case it should not contain the mode of parallelism. Why is future::plan(...) needed?

edit: and just to be explicit about the second part of my question - once _target.R defines a future::plan dynamic branching parallelism works out of the box.

[wlandau]

From the top of https://books.ropensci.org/targets/hpc.html#future-locally:

To parallelize targets over multiple processes on your local machine, declare a future plan in your _targets.R file. The callr plan from the future.callr package is recommended.

If you feel this is still unclear, feel free to submit a pull request to https://github.com/ropensci-books/targets.

When you run tar_make_future(), targets spins up a new external R process that sources _targets.R and then runs the pipeline. The purpose of _targets.R is to populate the global environment of the external R process and define the targets. The future::plan() needs to be defined in the global environment of that external process because that is where the parallel workers are launched. That is why the future::plan() needs to be in _targets.R.

[kkmann]

got it thanks!