Guidance, caveats or examples on how to integrate python/reticulate elements

[fBedecarrats]

Many thanks to the devs and contributors for this great package! I'm not sure if my question should be addressed to this repo or to the support book. I am looking for guidance, caveats or examples on how to build a Shiny app with {golem} that embeds a python model/application.
In short, we developed a python model (XGBoost) to forecast the number of kids that will come to have lunch in public schools to adapt the production and reduce food waste (available here). We use R for the data preparation and to run simulations, so we know how use {reticulate} to run the python model from R, pass arguments from R to the Python code and gather the ouputs from the python code to visualise them with {ggplot2}. Now we would like to build a Shiny app to facilitate the interactive use of the model by the colleagues in charge of organising the production. {golem} looks like a great canditate to ensure we follow development good practices to run into production. However, before I start the development, I still have many questions :

• Where should I put my python code to be consistent with {golem} app structure?
• Is there a way to automate the download of the python app I want to embed ad shiny app build to make sure to allways have an up-do-date version?
• What is the best way to handle the python environment variables (required packages, package source channels, virtual environment)?
• Are there some caveats to highlight regarding potential conflicts with dev functions provided by {golem}?

So this message is just to reach out to know if others know about examples of Shiny application using both {golem} and python/{reticulate} or, even better, some documented guidance on how to do it.

Cheers!

[fBedecarrats]

Referenced in #677. Should I close this one?

[ColinFay]

Hey, I'd prefer having it as a discussion, so let's keep this one open :)

[ruthkr]

Any updates on this? I've been following this guide with some modifications to adapt it to {golem}, which works great locally. However, I'm having issues in deploying my app to shinyapps.io.

A complete guide touching on the points mentioned by @fBedecarrats would be great indeed!

[fBedecarrats]

Hello @ruthkr , no updates on my side. I am also using the guide you cite to develop a shiny app that calls python scripts on the server side. My understanding is that the main trick (optional locally but mandatory on shinyapps) is to call reticulate::use_virtualenv before any other call to reticulate. I cannot figure out how to do that with golem, so I desisted from using it for my project (see repo here) . So far what I'm doing works fine locally with a Linux machine and on shinyupdates. I had to make some adjustments to .Rprofile to use it locally on Windows machines.

[ruthkr]

Hi @fBedecarrats,

I managed to use virtualenv by calling all the {reticulate} functions inside the server module, as shown below:

mod_modulename_server <- function(id) {
  moduleServer(id, function(input, output, session) {
    ns <- session$ns

    # Load virtualenv
    virtualenv_dir <- Sys.getenv("VIRTUALENV_NAME")
    python_path <- Sys.getenv("PYTHON_PATH")
    reticulate::use_python(python_path, required = TRUE)
    reticulate::use_virtualenv(virtualenv_dir, required = TRUE)
    
    # Load Python code
    reticulate::source_python("src/my_python_script.py")
    ...
  }
}

Hopefully this can help.

Since there's no correct/official guidelines, I'm doing some decisions such as putting the Python code in src (as one would do with {Rcpp} code, which may not be the best solution.

[fBedecarrats]

Thank you very much, it looks like an adequate approach to me. I will test that in a couple of weeks, as soon as I'm through my current project. Have you had the opportunity to test if this setup works to use golem built-in workflows produce docker image, packages or deploy on other shiny servers than shinyapps (shinyserver or Rstudio connect)?

[fBedecarrats]

Hello, other things came along the way, but I finally refactored my shiny app with Golem. I can confirm that the approach proposed by @ruthkr works. One should include extra_sysreqs = c("python3", "python3-pip", "python3-venv") in add_dockerfile() in the case of a docker deployment.
Many thanks @ruthkr for the solution!