Use targets with git submodules

[j-kreis]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I am working on multiple targets projects, which are implemented in separate git projects. My goal is to combine these projects in a single (parent) project using git submodules and then be able to use tar_read()/tar_make() in the main project, to access targets from the submodules.

I have read section 8.8 Multiple projects, which however assumes a shared folder for all the target scripts. I instead have projects with separate target scripts, that live in subdirectories.

├── _targets.yaml
├── _targets.R
├── R/
├──── functions_data.R
├──── functions_analysis.R
├──── functions_visualization.R
├──submodules
├──── project A
├──────  _targets.R
├──────  R/
├──────── functions_1.R
├──────── functions_2.R 
├──────  targets/
├──────── meta/
├──────── objects/
...
├──── project B
├──────  _targets.R
├──────  R/
├──────── functions_1.R
├──────── functions_2.R
├──────  targets/
├──────── meta/
├──────── objects/
...

Using Sys.setenv(TAR_PROJECT = "project A") and:

tar_config_set(store = "submodules/project A/_targets",
               project = "project A")

I am able to read data when I have used tar_make() within the submodules, but I am not able to use tar_make() to evaluate all targets of the parent project and the subprojects.

It would be possible, when I use the following configuration:

tar_config_set(script = "submodules/project A/_targets.R", 
               store = "submodules/project A/_targets",
               project = "project A")

But when tar_make() (called in the parent project) enters the subproject, it does not search scripts, that are sourced in submodules/project A/_targets.R in submodules/project A/R but in R.

Is there any way to set the root directory for a subproject?

[wlandau]

Depending on how you set up each _targets.R, you might have a call to tar_source() which sources the scripts in the R/ folder relative to the current directory. You could try supplying a custom files argument to point to either an absolute path or a path relative to a project root that you control (i.e. with the rprojroot package).

[j-kreis]

@wlandau thanks for you suggestion. The goal is actually, that all repositories still work independently (two independent projects that are referenced in a root project as chapters). So I don't want to set a relative (starting from /_targets.R) and can not set absolute paths. If project A or B are cloned, they are supposed to work independently, the root project just uses the targets of the subprojects. I tried to use here::here in submodules/project A/_targets.R, but could not figure out how to force sourcing the scripts in submodules/project A/R, not /R. here always starts at the root, not in submodules/project A/.

I tried the following setting, but here certainly will still start at the root directory. So I wondered if there is a parameter or option, so that the script parameter, that I set for the submodule is used.

submodules/project A/.here
submodules/project A/.Rproj

#> submodules/project A/_targets.R
tar_source(files = here::here('R'))

[wlandau]

Hmm... it seems like you would need to make a choice between running the individual submodules and running from the parent directory. It seems reasonable to run tar_make() in each submodule and then do something like withr::with_dir("submodule_a", tar_read(your_target)) when you want to read a target from one of the projects.

[hlageek]

I have a setting where I use the main project for orchestrating subprojects. If I would try to transpose my setting onto your problem, it would look something like this for the parent directory pipeline:

list(
 tar_target(
    name = submoduleA_path,
    command = here::here("submodules", "project A"),
    format = "file"
   ),
 tar_target(
    name = compiled_submoduleA,
    command = run_submodule(
        dir = submoduleA_path
       )
    )
)

where

run_submodule  <- function(dir){
     withr::with_dir(dir, targets::tar_make())
      return(Sys.time())
}

and downstream targets can be made dependent on compiled_submoduleA. @wlandau already suggested above how targets in submodules can be read.

Would something like this satisfy your usecase?

[j-kreis]

thanks that works perfectly, the only lines I had to change were repository = "local", and format = "file" in the second tar_target call.