Render directly to AWS s3?

[nerutenbeck]

Hi there,

Thanks for all your great work!

I'm wondering if it is possible to render an rmarkdown html report target directly to s3 without writing it locally. I've searched the docs but can't seem to find an example.

I'm a novice to targets and diving into a nested analytical pipeline right away. My use case is as follows:

• I want to build a targets pipeline that iterates over several case studies
• Each case study itself iterates over an analysis pipeline using data from a number of (standardized) data sources, and writes an intermediate analytical report for each data source
• Each case study then synthesizes the analytical output from the many data sources into a synthetic case study report.

To do this, I'm currently trying to begin by writing a complete target factory to operate on a single data source, which I will then map over all the data sources in a given case study, and then map that process over all case studies.

My basic data source factory looks something like this:

data_source_factory <- function(case_study, data_source) {
  tar_option_set(
    imports = c("myradpackage"),
    packages = c("glue", "tarchetypes"),
    resources = list(bucket = "my-rad-bucket", prefix = file.path(case_study, data_source))
  )
  list(
    tar_target_raw(
      "data_source",
      quote(prep_data(data_source)),
      format = "aws_qs"
    ),
    ..., # more rad analysis here
    tar_render_raw(
      "data_source_report",
      path = "path_to_my_rad_report.Rmd",
      render_arguments = quote(
        list(
          output_file = glue("{data_source}.html") # I want to send this directly to s3
        )
      )
    )
  )
}

There are many of these data source reports, and I would like to avoid rendering / tracking the reports locally, since I'm typically doing remote development on an EC2 instance, and when I terminate my instance the local targets will break and I will need to rerun the pipeline. One option is saving the intermediate reports in myradpackage and tracking them with version control, but as the number of data source reports increases, that will become annoying.

Hence my question:

1. Is it possible to specify something like format = aws_file for tar_render_raw, or is there a different recommended solution to achieve the same effect?
2. A side note - is there anything wrong with calling tar_option_set within a target factory?

It also occurs to me that the workflow I'm imagining here isn't optimal for my use case, so I would welcome advice on that front as well.

Thanks for your time.

[wlandau]

Interesting use case. I can see how the ability to send rendered reports to a Se would be appealing.

The "aws_file" format can only accept a single file, and it sends that file to a bucket at _targets/objects/target_name (no file extension). By contrast, tar_render() and tar_render_raw() return multiple files, including the source Rmd, which I gather you don't want whisked away to the cloud. So in your pipeline, I recommend including a aws_file target that copies the HTML file from the previous step and uploads the copy. Sketch:

tar_target_raw(
  data_upload,
  copy_html_to_tmp(data_source_report),
  format = "aws_file"
}

copy_html_to_tmp <- function(path) {
  tmp <- tempfile()
  html <- grep("html$", path, value = TRUE)
  file.copy(html, tmp)
  tmp
}

I would generally recommend against tar_option_set() in a target factory because it modifies global state. Could be okay though if you know for certain that you will not use other factories that modify target options.

[nerutenbeck]

Thanks for the clear advice! This idea occurred to me after I posted this question, but for my use case (developing on EC2) I gather in order to avoid breaking the target tracking because of remote development I would also want to skip tracking updates to the output of tar_render_raw, then, also, because otherwise I will need to either carry the output around in myradpackage or rerun the pipeline in any new development environment.

[wlandau]

I think the source Rmd can live in your package, but I would recommend keeping the HTML output somewhere else, regardless of where you run the pipeline. That goes for the _targets/ data store as well. You might be able to keep it on the persistent volume of the EC2 instance, Git LFS, or even a bucket if you pull/sync right before you run the pipeline.

[nerutenbeck]

Ah, that is an interesting idea - I like the idea of syncing the _targets/ data store to the relevant bucket in s3 and syncing down / up when I need to rerun the pipeline. I take it that these are things I will need to do outside of _targets.R, e.g in the readme or a separate executable like so:

# build bash commands
s3_path <- "s3://myradbucket/_targets/"
local_path <- ".../myradpackage/_targets/"

sync_down <- glue::glue("aws s3 sync {s3_path} {local_path}")
sync_up <- glue::glue("aws s3 sync {local_path} {s3_path}")

# execute pipeline
system(sync_down)
targets::tar_make()
system(sync_up)

[wlandau]

Yeah, that's exactly what I had in mind. Syncing should be done before and after running the pipeline.