[help] scratch/ directory disappearing when working in a different R session?

[billdenney]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I have a few questions:

• Is it allowed by targets to run tar_make() in multiple R sessions simultaneously? (It's okay if the answer is no, and I would guess that it's not preferred, regardless.)
• Is it allowed by targets to run tar_read() in a separate R process while tar_make() is being run?
• Related to the questions below, is it a reasonable guess that the scratch directory may be removed when working in a separate process? And, would it be reasonable for targets to leave the scratch directory in place rather than removing it?

My guess is that the overall answer will be, "Bill, don't do that." My rationale for wanting to at least do a tar_read() in another process is to understand some of the ongoing results and make new targets for the next tar_make() run. (For example, I'm making a complex model, I look at one model's results to make changes for what I want to explore in the next model.)

I tried to make a reprex, but my tries to this point are not successful.

With some of my long-running targets pipelines, I sometimes may want to look at some other objects while the pipeline is running in another R process.

Sometimes, when the pipeline running the main jobs comes back, I'll get an error like the following:

▶ end pipeline [2.749 hours]
Warning message:
In gzfile(file, mode) :
  cannot open compressed file '_targets/scratch/[targetname]', probable reason 'No such file or directory'
Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: _store_ cannot open the connection

It seems like the scratch directory no longer exists, and I think that may be due to the fact that I'm looking at objects in another R process. I seem to see the issue happening more often if I run tar_make() in the other process. I think that I may be violating one of the assumptions of targets when I run tar_make() in another process, but it seems like it should be okay to do a tar_read() in the other process.

[wlandau]

tar_read() can be run in a separate process while the pipeline is running, but be careful not to load a target that is currently saving or building because then there could be file conflicts. There can only be one instance of a pipeline executing (tar_make()) at a time.