Advice for including shell script that downloads data

[Aariq]

I have a workflow that starts with downloading climate data using a shell script. The shell script checks whether the data exists already, verifies the checksum, and then only downloads the data if it doesn't already exist. That script downloads many files. I'm not sure how to include that in a targets workflow. I can write an R wrapper to run the script, but since it already checks if the files are outdated, would it even make sense to try to include it in tar_plan()? Should I just put that wrapper function in _targets.R, but outside of tar_plan()?

[wlandau]

You could write a target like this:

tar_target(
  files, {
    system2("./your_script.sh")
    c("your_script.sh", "output_file1.csv", "output_file2.csv"),
    format = "file"
  }
)

File targets are documented at https://books.ropensci.org/targets/files.html#external-input-files and https://books.ropensci.org/targets/files.html#external-output-files. targets does not care about the difference between input and output files, it only knows you have a format = 'file" target that identifies files to track after some work is done. So you could write a target that does the work and then returns a character vector of input and output file paths.

A little redundancy for file tracking is not usually a problem, but if the script only downloads the data if the file does not already exist, it may fail to download the most up-to-date versions of those files. Likewise with ordinary file targets. So maybe consider format = "url" for your use case if each file has its own valid URL? If not, maybe relax the file checking in the shell script and consider one of the helpers at https://docs.ropensci.org/tarchetypes/reference/index.html#section-targets-with-custom-invalidation-rules or https://docs.ropensci.org/tarchetypes/reference/index.html#section-cues to customize when files are downloaded.