compile problem with use of update.brmsfit in targets pipeline

[hansvancalster]

I'm trying to figure out an easy way to use brms in a targets pipeline. I need to fit lots of the same models using different datasets. My strategy is to use brms::update.brmsfit() for this purpose to avoid recompiling the model. I first fit an 'empty' model using 'chains = 0' and then refit the model using brms::update.brmsfit() with the new data.
This works fine when you run tar_make() for the first time, but if you then delete the fitted model from the store and run tar_make() again, we get 'Error : Model not compiled. Try running the compile() method first'.

NB: I am using 'backend = "cmdstanr"'. The problem does not reproduce when using 'backend = "rstan"' (but I have another problem with 'backend = "rstan"' for which I will start another discussion).

Any suggestions how to solve this problem?

targets::tar_dir({ # tar_dir() runs code from a temporary directory.
  targets::tar_script({
    library(targets)
    tar_option_set(packages = c("brms", "dplyr"))
    list(
      tar_target(raw_data, airquality),
      tar_target(data, raw_data %>% filter(!is.na(Ozone))),
      tar_target(empty_fit,
                 brm(Ozone ~ Wind + Temp, data, chains = 0,
                     backend = "cmdstanr")),
      tar_target(fit,
                 update(empty_fit,
                        chains = 4,
                        refresh = 0,
                        silent = 2))
    )
  })
  targets::tar_make()
  targets::tar_delete(fit)
  targets::tar_make()
})
#> * start target raw_data
#> * built target raw_data
#> * start target data
#> * built target data
#> * start target empty_fit
#> Start sampling
#> Running MCMC with 1 chain...
#> 
#> Chain 1 WARNING: No variance estimation is 
#> Chain 1          performed for num_warmup < 20 
#> Chain 1 Iteration: 1 / 2 [ 50%]  (Warmup) 
#> Chain 1 Iteration: 2 / 2 [100%]  (Sampling) 
#> Chain 1 finished in 0.0 seconds.
#> 
#> Warning: 1 of 1 (100.0%) transitions ended with a divergence.
#> This may indicate insufficient exploration of the posterior distribution.
#> Possible remedies include: 
#>   * Increasing adapt_delta closer to 1 (default is 0.8) 
#>   * Reparameterizing the model (e.g. using a non-centered parameterization)
#>   * Using informative or weakly informative prior distributions 
#> 
#> * built target empty_fit
#> * start target fit
#> Running MCMC with 4 sequential chains...
#> 
#> Chain 1 finished in 0.2 seconds.
#> Chain 2 finished in 0.2 seconds.
#> Chain 3 finished in 0.2 seconds.
#> Chain 4 finished in 0.2 seconds.
#> 
#> All 4 chains finished successfully.
#> Mean chain execution time: 0.2 seconds.
#> Total execution time: 1.2 seconds.
#> * built target fit
#> * end pipeline
#> v skip target raw_data
#> v skip target data
#> v skip target empty_fit
#> * start target fit
#> x error target fit
#> * end pipeline
#> Error : Model not compiled. Try running the compile() method first.
#> Error: callr subprocess failed: Model not compiled. Try running the compile() method first.
#> Visit https://books.ropensci.org/targets/debugging.html for debugging advice.

^{Created on 2021-10-07 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value                       
#>  version  R version 4.1.1 (2021-08-10)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  Dutch_Belgium.1252          
#>  ctype    Dutch_Belgium.1252          
#>  tz       Europe/Paris                
#>  date     2021-10-07                  
#> 
#> - Packages -------------------------------------------------------------------
#>  package     * version    date       lib
#>  callr         3.7.0      2021-04-20 [1]
#>  cli           3.0.1      2021-07-17 [1]
#>  codetools     0.2-18     2020-11-04 [2]
#>  crayon        1.4.1      2021-02-08 [1]
#>  data.table    1.14.0     2021-02-21 [1]
#>  digest        0.6.28     2021-09-23 [1]
#>  ellipsis      0.3.2      2021-04-29 [1]
#>  evaluate      0.14       2019-05-28 [1]
#>  fansi         0.5.0      2021-05-25 [1]
#>  fastmap       1.1.0      2021-01-25 [1]
#>  fs            1.5.0      2020-07-31 [1]
#>  glue          1.4.2      2020-08-27 [1]
#>  highr         0.9        2021-04-16 [1]
#>  htmltools     0.5.2      2021-08-25 [1]
#>  igraph        1.2.6      2020-10-06 [1]
#>  knitr         1.36       2021-09-29 [1]
#>  lifecycle     1.0.1      2021-09-24 [1]
#>  magrittr      2.0.1      2020-11-17 [1]
#>  pillar        1.6.3      2021-09-26 [1]
#>  pkgconfig     2.0.3      2019-09-22 [1]
#>  processx      3.5.2      2021-04-30 [1]
#>  ps            1.6.0      2021-02-28 [1]
#>  purrr         0.3.4      2020-04-17 [1]
#>  R6            2.5.1      2021-08-19 [1]
#>  reprex        2.0.1      2021-08-05 [1]
#>  rlang         0.4.11     2021-04-30 [1]
#>  rmarkdown     2.11       2021-09-14 [1]
#>  rstudioapi    0.13       2020-11-12 [1]
#>  sessioninfo   1.1.1      2018-11-05 [1]
#>  stringi       1.7.4      2021-08-25 [1]
#>  stringr       1.4.0      2019-02-10 [1]
#>  targets       0.8.0.9000 2021-09-29 [1]
#>  tibble        3.1.4      2021-08-25 [1]
#>  tidyselect    1.1.1      2021-04-30 [1]
#>  utf8          1.2.2      2021-07-24 [1]
#>  vctrs         0.3.8      2021-04-29 [1]
#>  withr         2.4.2      2021-04-18 [1]
#>  xfun          0.26       2021-09-14 [1]
#>  yaml          2.2.1      2020-02-01 [1]
#>  source                                   
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  https://ropensci.r-universe.dev (R 4.1.1)
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#> 
#> [1] C:/R/library
#> [2] C:/R/R-4.1.1/library

[stuvet]

I tried to overcome the same problem for a while. I definitely managed it at some point, but I'm unsure whether I've just accepted recompilation with my current code or not EDIT: I've just checked - it doesn't recompile, even when run on a worker. This was all a few months back, so apologies if I get any of it wrong.

Root causes:

1. From memory, I believe brms::brm saves the compiled model in /tmp/... & inserts a reference to the file in the resulting brmsfit. After the tempfile has been removed (or if the directory is not mounted to workers) the model recompiles even when update is called. Perhaps @paul-buerkner can confirm/refute?
2. It seemed that some update.brmsfit options, if missing, trigger recompilation.
3. Although it doesn't look like it in your code, I'd guess that passing newdata & using any default priors could potentially result in recompilation, depending on the priors brms chose.

Attempted solution

I tried using the file & save_model arguments of brms::brm to save the brmsfit & compiled models in accessible places. I believe I also had to set stan_model_args = list(dir = 'my_dir', stanc_options = list(o = 'my_model')) (where my_dir = dirname(save_model) & my_model = basename(save_model)) to make sure the stan model returned into the brmsfit had the correct reference.

I also spent some time digging through the source code to try to work out what arguments needed to be passed to update.brmsfit to avoid recompilation. I settled on passing newdata & the same chains, iter, warmup, control & init as when the model was first created. I also passed recompile = ...file_is_absent().

I'm not certain that my code is avoiding recompilation at the moment as I've changed a few things since, but certain I managed it at some point - this might get you along the right path unless others offer more concrete suggestions.

[hansvancalster]

Thanks @stuvet, this has been very helpful 👍

I tried your suggestion and confirm that it works if you specify file, save_model and stan_model_args in the first call to brms::brm(). In subsequent update() statements the model is not recompiled even though I did change chains from 0 to 4 (this is evidenced by the timings at the end of the reprex).

targets::tar_dir({ # tar_dir() runs code from a temporary directory.
  targets::tar_script({
    library(targets)
    tar_option_set(packages = c("brms", "dplyr"),
                   format = "qs")
    list(
      tar_target(raw_data, airquality),
      tar_target(data, raw_data %>% filter(!is.na(Ozone))),
      tar_target(empty_fit,
                 brm(Ozone ~ Wind + Temp,
                     data,
                     chains = 0,
                     backend = "cmdstanr",
                     file = "./empty_fit",
                     save_model = "./empty_fit",
                     stan_model_args = list(
                       dir = ".",
                       stanc_options = list(o = "empty_fit")
                       )
                     )
                 ),
      tar_target(fit,
                 update(empty_fit,
                        chains = 4,
                        refresh = 0,
                        silent = 2)),
      tar_target(fit_newdata,
                 update(empty_fit,
                        newdata = data[1:100, ],
                        chains = 4,
                        refresh = 0,
                        silent = 2))
    )
  })
  targets::tar_make()
  targets::tar_delete(fit)
  targets::tar_make()
  targets::tar_meta(fields = "seconds")
})
#> * start target raw_data
#> * built target raw_data
#> * start target data
#> * built target data
#> * start target empty_fit
#> Start sampling
#> Running MCMC with 1 chain...
#> 
#> Chain 1 WARNING: No variance estimation is 
#> Chain 1          performed for num_warmup < 20 
#> Chain 1 Iteration: 1 / 2 [ 50%]  (Warmup) 
#> Chain 1 Iteration: 2 / 2 [100%]  (Sampling) 
#> Chain 1 finished in 0.0 seconds.
#> 
#> Warning: 1 of 1 (100.0%) transitions ended with a divergence.
#> This may indicate insufficient exploration of the posterior distribution.
#> Possible remedies include: 
#>   * Increasing adapt_delta closer to 1 (default is 0.8) 
#>   * Reparameterizing the model (e.g. using a non-centered parameterization)
#>   * Using informative or weakly informative prior distributions 
#> 
#> * built target empty_fit
#> * start target fit
#> Running MCMC with 4 sequential chains...
#> 
#> Chain 1 finished in 0.2 seconds.
#> Chain 2 finished in 0.2 seconds.
#> Chain 3 finished in 0.2 seconds.
#> Chain 4 finished in 0.2 seconds.
#> 
#> All 4 chains finished successfully.
#> Mean chain execution time: 0.2 seconds.
#> Total execution time: 1.2 seconds.
#> * built target fit
#> * start target fit_newdata
#> Running MCMC with 4 sequential chains...
#> 
#> Chain 1 finished in 0.2 seconds.
#> Chain 2 finished in 0.2 seconds.
#> Chain 3 finished in 0.2 seconds.
#> Chain 4 finished in 0.2 seconds.
#> 
#> All 4 chains finished successfully.
#> Mean chain execution time: 0.2 seconds.
#> Total execution time: 1.2 seconds.
#> * built target fit_newdata
#> * end pipeline
#> v skip target raw_data
#> v skip target data
#> v skip target empty_fit
#> * start target fit
#> Running MCMC with 4 sequential chains...
#> 
#> Chain 1 finished in 0.2 seconds.
#> Chain 2 finished in 0.2 seconds.
#> Chain 3 finished in 0.2 seconds.
#> Chain 4 finished in 0.2 seconds.
#> 
#> All 4 chains finished successfully.
#> Mean chain execution time: 0.2 seconds.
#> Total execution time: 1.2 seconds.
#> * built target fit
#> v skip target fit_newdata
#> * end pipeline
#> # A tibble: 5 x 2
#>   name        seconds
#>   <chr>         <dbl>
#> 1 raw_data       2.53
#> 2 data           0   
#> 3 empty_fit     28.8 
#> 4 fit_newdata   13.1 
#> 5 fit           13.4

^{Created on 2021-10-07 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value                       
#>  version  R version 4.1.1 (2021-08-10)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  Dutch_Belgium.1252          
#>  ctype    Dutch_Belgium.1252          
#>  tz       Europe/Paris                
#>  date     2021-10-07                  
#> 
#> - Packages -------------------------------------------------------------------
#>  package     * version    date       lib
#>  callr         3.7.0      2021-04-20 [1]
#>  cli           3.0.1      2021-07-17 [1]
#>  codetools     0.2-18     2020-11-04 [2]
#>  crayon        1.4.1      2021-02-08 [1]
#>  data.table    1.14.0     2021-02-21 [1]
#>  digest        0.6.28     2021-09-23 [1]
#>  ellipsis      0.3.2      2021-04-29 [1]
#>  evaluate      0.14       2019-05-28 [1]
#>  fansi         0.5.0      2021-05-25 [1]
#>  fastmap       1.1.0      2021-01-25 [1]
#>  fs            1.5.0      2020-07-31 [1]
#>  glue          1.4.2      2020-08-27 [1]
#>  highr         0.9        2021-04-16 [1]
#>  htmltools     0.5.2      2021-08-25 [1]
#>  igraph        1.2.6      2020-10-06 [1]
#>  knitr         1.36       2021-09-29 [1]
#>  lifecycle     1.0.1      2021-09-24 [1]
#>  magrittr      2.0.1      2020-11-17 [1]
#>  pillar        1.6.3      2021-09-26 [1]
#>  pkgconfig     2.0.3      2019-09-22 [1]
#>  processx      3.5.2      2021-04-30 [1]
#>  ps            1.6.0      2021-02-28 [1]
#>  purrr         0.3.4      2020-04-17 [1]
#>  R6            2.5.1      2021-08-19 [1]
#>  reprex        2.0.1      2021-08-05 [1]
#>  rlang         0.4.11     2021-04-30 [1]
#>  rmarkdown     2.11       2021-09-14 [1]
#>  rstudioapi    0.13       2020-11-12 [1]
#>  sessioninfo   1.1.1      2018-11-05 [1]
#>  stringi       1.7.4      2021-08-25 [1]
#>  stringr       1.4.0      2019-02-10 [1]
#>  targets       0.8.0.9000 2021-09-29 [1]
#>  tibble        3.1.4      2021-08-25 [1]
#>  tidyselect    1.1.1      2021-04-30 [1]
#>  utf8          1.2.2      2021-07-24 [1]
#>  vctrs         0.3.8      2021-04-29 [1]
#>  withr         2.4.2      2021-04-18 [1]
#>  xfun          0.26       2021-09-14 [1]
#>  yaml          2.2.1      2020-02-01 [1]
#>  source                                   
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  https://ropensci.r-universe.dev (R 4.1.1)
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.0)                           
#>  CRAN (R 4.1.1)                           
#>  CRAN (R 4.1.0)                           
#> 
#> [1] C:/R/library
#> [2] C:/R/R-4.1.1/library

[stuvet]

Ah yes, of course I also changed the chains argument - that's the whole point! I just read off the ones I was specifying the second time around. Glad it helped, as I remember this one being frustrating to try to track down.