Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile model methods once and reuse for all models #894

Open
wants to merge 25 commits into
base: master
Choose a base branch
from

Conversation

andrjohns
Copy link
Collaborator

@andrjohns andrjohns commented Jan 10, 2024

Submission Checklist

  • Run unit tests
  • Declare copyright holder and agree to license (see below)

Summary

This PR adds the ability for the model methods to be pre-compiled and then simply linked to the object file produced by cmdstan. This will significantly speed up users' workflows, as they only need to perform the model method compilation once, instead of every time $init_model_methods() is called

Copyright and Licensing

Please list the copyright holder for the work you are submitting
(this will be you or your assignee, such as a university or company):
Andrew Johnson

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses:

@codecov-commenter
Copy link

codecov-commenter commented Jan 11, 2024

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (3c7a1a9) 88.28% compared to head (eae8189) 88.32%.

❗ Current head eae8189 differs from pull request most recent head 0ac09f9. Consider uploading reports for the commit 0ac09f9 to get more accurate results

Files Patch % Lines
R/utils.R 93.05% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #894      +/-   ##
==========================================
+ Coverage   88.28%   88.32%   +0.03%     
==========================================
  Files          12       12              
  Lines        4534     4592      +58     
==========================================
+ Hits         4003     4056      +53     
- Misses        531      536       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@avehtari
Copy link
Contributor

I tested as

library(cmdstanr)
# Using a non-merged PR #894
devtools::load_all('~/proj/cmdstanr2')
model1 <- cmdstan_model(stan_file = root("Birthdays", "gpbf1.stan"),
                        include_paths = root("Birthdays"),
                        compile_model_methods=TRUE, force_recompile=TRUE)

and got

Compiling Stan program...
Compiling and caching additional model methods...
Linking precompiled model methods to model object file...
Error in dyn.load(methods_dll, local = TRUE, now = TRUE) : 
  unable to load shared object '/tmp/RtmpNbmTDV/file2c1322e092b0.so':
  /tmp/RtmpNbmTDV/file2c1322e092b0.so: cannot open shared object file: No such file or directory

Can you provide more instructions on how to test this?

@andrjohns
Copy link
Collaborator Author

@avehtari Thanks for catching that issue! It looks like Linux needs the Stan model to be compiled with an additional flag for R to be able to link to it afterwards.

I've pushed a fix and the CI is passing, can you pull the changes and try again when you get a minute?

@avehtari
Copy link
Contributor

I get the same error

@jgabry
Copy link
Member

jgabry commented Jan 23, 2024

This will significantly speed up users' workflows, as they only need to perform the model method compilation once, instead of every time $init_model_methods() is called

Awesome! Thanks for working on this.

@andrjohns andrjohns changed the title [WIP] Compile model methods once and reuse for all models Compile model methods once and reuse for all models Jan 24, 2024
@andrjohns
Copy link
Collaborator Author

Sorted! This is ready for review now

@avehtari
Copy link
Contributor

Compilation and re-use of model methods in other model worked.

But it seems I still need to recompile every model in a session, and I just save the time not needing to recompile the model methods part several times? As the models usually don't need to be recompiled, can you explain why the model methods can't be used with pre-compiled models?

@avehtari
Copy link
Contributor

Compiling brms generated model with compile_model_methd=TRUE crashes. Also with another brms model, but not with my own handwritten model code.

library(cmdstanr)
data("VerbAgg", package = "lme4")
VerbAgg$r3 <- as.numeric(VerbAgg$resp)
sc <- brms::make_stancode(r3 ~ btype + mode + situ + (btype + mode + situ | id), 
                    data = VerbAgg, family = brms::cumulative())
sf <- write_stan_file(sc)
m1 <- cmdstan_model(sf, compile_model_methods=TRUE, force_recompile=TRUE)
Linking precompiled model methods to model object file...

 *** caught segfault ***
address 0x7f0b7bb8c008, cause 'invalid permissions'

Traceback:
 1: dyn.load(methods_dll, local = TRUE, now = TRUE)
 2: force(code)
 3: force(code)
 4: with_envvar(c(R_MAKEVARS_USER = makevars_file), {    set_makevars(new, path, makevars_file, assignment = assignment)    force(code)})
 5: withr::with_makevars(new_makevars, expr)
 6: force(code)
 7: withr::with_path(c(paste0(cmdstan_path(), lib_paths), toolchain_PATH_env_var()),     withr::with_makevars(new_makevars, expr))
 8: with_cmdstan_flags(dyn.load(methods_dll, local = TRUE, now = TRUE))
 9: expose_model_methods(private$model_methods_env_, verbose = !quiet)
10: self$compile(...)
11: initialize(...)
12: CmdStanModel$new(stan_file = stan_file, exe_file = exe_file,     compile = compile, ...)
13: cmdstan_model(sf, compile_model_methods = TRUE, force_recompile = TRUE)

@andrjohns
Copy link
Collaborator Author

Looks related to this: https://discourse.mc-stan.org/t/segfault-when-using-brms-cmdstanr-compile-model-methods-true/33771/4

Can you try compiling with STAN_THREADS enabled?

@avehtari
Copy link
Contributor

Another error

> + > > Compiling Stan program...
 
 *** caught segfault ***
address 0x7f6676099008, cause 'invalid permissions'

Traceback:
 1: dyn.load("/tmp/RtmpmAozHV/sourceCpp-x86_64-pc-linux-gnu-1.0.12/sourcecpp_272bf5990ba0/sourceCpp_2.so")
 2: eval(ei, envir)
 3: eval(ei, envir)
 4: withVisible(eval(ei, envir))
 5: source(scriptPath, local = env)
 6: Rcpp::sourceCpp(code = code, env = env, verbose = verbose)
 7: force(code)
 8: force(code)
 9: with_envvar(c(R_MAKEVARS_USER = makevars_file), {    set_makevars(new, path, makevars_file, assignment = assignment)    force(code)})
10: withr::with_makevars(c(USE_CXX14 = 1, PKG_CPPFLAGS = ifelse(cmdstan_version() <=     "2.30.1", "-DCMDSTAN_JSON", ""), PKG_CXXFLAGS = paste0(cxxflags,     cmdstanr_includes, collapse = " "), PKG_LIBS = libs), Rcpp::sourceCpp(code = code,     env = env, verbose = verbose))
11: force(code)
12: withr::with_path(paste0(cmdstan_path(), lib_paths), withr::with_makevars(c(USE_CXX14 = 1,     PKG_CPPFLAGS = ifelse(cmdstan_version() <= "2.30.1", "-DCMDSTAN_JSON",         ""), PKG_CXXFLAGS = paste0(cxxflags, cmdstanr_includes,         collapse = " "), PKG_LIBS = libs), Rcpp::sourceCpp(code = code,     env = env, verbose = verbose)))
13: rcpp_source_stan(code, env, verbose)
14: expose_model_methods(env = private$model_methods_env_, verbose = !quiet,     hessian = compile_hessian_method)
15: self$compile(...)
16: initialize(...)
17: CmdStanModel$new(stan_file = stan_file, exe_file = exe_file,     compile = compile, ...)
18: cmdstan_model(sf, compile_model_methods = TRUE, force_recompile = TRUE)

@andrjohns
Copy link
Collaborator Author

That call stack indicates that you're not on this branch, since it's using the "old" pathway for exposing model methods

@andrjohns
Copy link
Collaborator Author

Either way I'll have a proper look into this tomorrow and add a fix

@avehtari
Copy link
Contributor

After rebuilding with threads I get

> + + > > Compiling Stan program...
Linking precompiled model methods to model object file...
Error in dyn.load(methods_dll, local = TRUE, now = TRUE) : 
  unable to load shared object '/tmp/RtmpS8Bk6H/file282fa157edb31.so':
  /tmp/RtmpS8Bk6H/file282fa157edb31.so: cannot open shared object file: No such file or directory

@andrjohns
Copy link
Collaborator Author

FYI that I'm going to leave this PR for the v1.0/CRAN branch. A lot of the complexity/issues here are caused by the windows cmdstan using mingw32-make/gcc from pacman, while R and Rcpp use the RTools utilities, causing a bunch of headaches when linking objects between the two.

Once we move to RTools-only on windows, this PR/implementation will be much simpler and easier

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants