managing packages on HPC

[Aariq]

If I'm using targets with clustermq via SSH, do I need to manually install packages on the HPC I'm connecting to? If so, is there a way to do this automatically with renv? I know this might be out of the scope of targets, but it would be nice if there was a way to ensure that the HPC has the same versions of packages installed locally for consistency.

[wlandau]

Yes, when you use clustermq::Q() or targets::tar_make_clustermq(), the workers need to be running the same version of R with the same versions of packages as the local main process. The way you do this depends on the HPC system you have. At workplace, we have a traditional Univa/Sun Grid Engine (SGE) scheduler backed by a network file system (NFS). This ensures that any packages I install locally will automatically be available to parallel workers. Other systems may be similar, but I unfortunately do not know enough to give specific advice that will fully generalize. This is a good conversation to have with your sys admin.

[Aariq]

My University HPC uses SLURM and I'm SSHing in from my personal laptop, so I don't think there is any built-in way for the HPC to know what packages I have installed locally. I'm running tar_make_clustermq() locally on my laptop, as I only want some targets to run on the HPC (e.g. I want models to run on HPC in parallel, but I want to knit .Rmd files locally). I was trying to figure out if there was some way to send my renv lockfile to the HPC and have it run renv::restore(). Is that something that could be added to a SLURM template or something?

[wlandau]

I think you have some degree of control through the template file you supply to options(clustermq.template = "..."). You could try starting with https://github.com/mschubert/clustermq/blob/master/inst/SLURM.tmpl. Sketch:

#!/bin/sh
#SBATCH --job-name={{ job_name }}
#SBATCH --output={{ log_file | /dev/null }}
#SBATCH --error={{ log_file | /dev/null }}
#SBATCH --mem-per-cpu={{ memory | 4096 }}
#SBATCH --array=1-{{ n_jobs }}
#SBATCH --cpus-per-task={{ cores | 1 }}

cd /directory/with/local/lib
Rscript -e 'renv::restore()'
CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker("{{ master }}")'

[Aariq]

Thanks, this actually helps a lot. I think I know how to do this now.