Difficulty with targets + future.batchtools on HPC w/ Slurm

[mbrousil]

Hello! @mishafredmeyer and I been struggling for a while to get a minimal targets + future.batchtools workflow to run as intended on our institution's HPC (Slurm). I'm hoping that someone can help us find where we're going wrong.

With our current setup the targets workflow seems to be running, but the secondary processes are not sent to the partition that we specify. They seem to just be going to the backfill partition on the cluster. Because of this I assume that other parts of the template are not being implemented as intended. What do we need to do to make this run properly?

The latest run provided the following .out content from one of the secondary processes:

### [bt]: This is batchtools v0.9.15
### [bt]: Starting calculation of 1 jobs
### [bt]: Setting working directory to '/home/myusername/targets_test'
Loading required package: targets
### [bt]: Memory measurement disabled
### [bt]: Starting job [batchtools job.id=1]
### [bt]: Setting seed to 23242 ...

### [bt]: Job terminated successfully [batchtools job.id=1]
### [bt]: Calculation finished!

At the moment we have the following file structure:

+-- batchtools.slurm.tmpl
+-- run_workflow.R
+-- run_workflow.sh
+-- _targets
|   +-- meta
|   \-- objects
\-- _targets.R

Our scripts look like this:

run_workflow.sh:

#!/bin/bash
#SBATCH --partition=mypartition
#SBATCH --job-name=targets_test
#SBATCH -o targets_test.out
#SBATCH --time=0-00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --mail-type=ALL
#SBATCH --mail-user=myusername@email
#SBATCH --get-user-env
#SBATCH --verbose
#SBATCH --mem=2048

# Load R on compute node
module load r/4.0.2

# Working dir
echo $PWD

echo "Run script run_workflow.R"

Rscript --vanilla run_workflow.R

run_workflow.R:

library(targets)
library(future)

tar_destroy()

tar_make_future(
  workers = 2)

_targets.R:

library(targets)
library(future)
library(future.batchtools)
library(tidyverse)

plan(strategy = batchtools_slurm,
     template = "batchtools.slurm.tmpl",
     resources = list(
       partition = 'mypartition',
       walltime = 600,
       ncpus = 1,
       measure.memory = TRUE,
       user_email = 'myusername@email',
       memory = '16gb'
     ))


list(

  tar_target(the_data, tibble(mtcars)),
  tar_target(model_it, lm(mpg ~ cyl, data = the_data)),
  tar_target(figure, ggplot(data = the_data) + geom_point(aes(x = cyl, y = mpg)))
)

.batchtools.conf.R:

cluster.functions <- makeClusterFunctionsSlurm(template = "batchtools.slurm.tmpl")

batchtools.slurm.tmpl:

#!/bin/bash

## Job Resource Interface Definition
##
## ntasks [integer(1)]:       Number of required tasks,
##                            Set larger than 1 if you want to further parallelize
##                            with MPI within your job.
## cpus-per-task [integer(1)]: Number of required cpus per task,
##                            Set larger than 1 if you want to further parallelize
##                            with multicore/parallel within each task.
## walltime [integer(1)]:     Set a limit on the total run time of the job allocation.
##                            Must be at least 60 seconds.
## memory   [integer(1)]:     Memory in megabytes required per node.
##
## Default resources can be set in your .batchtools.conf.R by defining the variable
## 'default.resources' as a named list.

cd /home/myusername/targets_test

<%
# relative paths are not handled well by Slurm
log.file = fs::path_expand(log.file)
-%>

#SBATCH --job-name=<%= job.name %>
#SBATCH --output=<%= log.file %>
#SBATCH --error=<%= log.file %>
#SBATCH --time=<%= ceiling(resources$walltime / 60) %>
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<%= resources$ncpus %>
#SBATCH --mem=<%= resources$memory %>
#SBATCH --mail-type=ALL
#SBATCH --mail-user=<%= resources$user_email %>
#SBATCH --verbose
<%= if (!is.null(resources$partition)) sprintf(paste0("#SBATCH --partition='", resources$partition, "'")) %>
<%= if (array.jobs) sprintf("#SBATCH --array=1-%i", nrow(jobs)) else "" %>

module load R/4.0.2

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

## Run R:
## we merge R output with stdout from SLURM, which gets then logged via --output option
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

Thanks for your help!

[wlandau]

I recommend peeling back some layers. Do the jobs still go to the wrong partition if you invoke future()/future.batchtools without targets? If so, batchtools.slurm.tmpl may need fixing, and this is easier to work on outside a targets pipeline. And what happens if you hard code the partition in batchtools.slurm.tmpl without going through resources?

[mbrousil]

Thanks! This led me to something that worked. For reference, I did both steps that you suggested:

And what happens if you hard code the partition in batchtools.slurm.tmpl without going through resources?

I hard coded it in to the .tmpl and removed the specification from the _targets.R script. It still went through the backfill.

Do the jobs still go to the wrong partition if you invoke future()/future.batchtools without targets? If so, batchtools.slurm.tmpl may need fixing, and this is easier to work on outside a targets pipeline.

Yep, they still did. I then worked within this batchtools-only version and got a new .tmpl file from here and this ended up working. Not sure why, though. I've been able to add targets back in now and run a subset of the larger workflow I'm building with some success.

On a related note, if I wanted the submission job (run_workflow.sh) to have a different walltime than each of the jobs produced using batchtools.slurm.tmpl, would I need to do something specific to make this work? Over the weekend I seemed to be running into issues when the walltimes were not identical across the board.

Thank you for your help!

[wlandau]

Yep, they still did. I then worked within this batchtools-only version and got a new .tmpl file from here and this ended up working. Not sure why, though. I've been able to add targets back in now and run a subset of the larger workflow I'm building with some success.

Glad to hear it. I do not know SLURM personally well enough to guess what could have fixed it, but starting fresh with a template that works is definitely a solid tactic. Michel, the maintainer of batchtools, would know better than I would.

On a related note, if I wanted the submission job (run_workflow.sh) to have a different walltime than each of the jobs produced using batchtools.slurm.tmpl, would I need to do something specific to make this work? Over the weekend I seemed to be running into issues when the walltimes were not identical across the board.

From your earlier template file, it looks like walltime is a parameter

#SBATCH --time=<%= ceiling(resources$walltime / 60) %>

If you want to keep using variable walltimes instead of hard-coding a single value, it might be good to encode a default, either through tar_option_set(resources = list(walltime = ...)) in _targets.R or directly in the template file.

#SBATCH --time=<%= if (is.null(resources$walltime)) 120 else ceiling(resources$walltime / 60) %>

Non-default values get supplied to resources through tar_target(..., resources = list(walltime = ...)). Supplying a resources list to an individual target will overwrite all the other default resources. For example, for the target in the _targets.R file below, resources$ncpus will be NULL.

# _targets.R
library(targets)
library(future.batchtools)
plan(strategy = batchtools_slurm, template = "batchtools.slurm.tmpl")
tar_option_set(resources = list(ncpus = 2, walltime = 120))
list(tar_target(x, your_command(), resources = list(walltime = 240))) # should also supply ncpus here if the template file has no default.

[wlandau]

Also, I only just noticed that in your original _targets.R, you supplied resources to plan(). I recommend supplying it instead to tar_options_set() and/or individual targets.

[c1au6i0]

I have been struggling to transition from {clustermq} to {future.batchtools} in Slurm and so I tried to following the steps of @mbrousil .
I used the same template of batchtools.slurm.tmpl in the directory of the project and same .batchtools.conf.R that I placed in my home directory.
In my _targets.R:

plan(strategy = batchtools_slurm,
     template = "batchtools.slurm.tmpl"
)
resources_all <- tar_resources(
  clustermq = tar_resources_clustermq(template = list(n_cores = 1, memory = 150000)),
  future = tar_resources_future(resource = list(ncpus = 1, memory = 16000, walltime = 600, partition = 'scu_cpu'))
tar_option_set(resources = resources_all)
)

In my case it seems that I cannot successfully define sbatch options in resources. If I hard code the option in the "batchtools.slurm.tmpl" it works otherwise I get something like this depending on the option that I set on resources (ncpus in this example):

'sbatch: error: Invalid numeric value "" for --cpus-per-task.'

If I hardcode everything and define the partition in resources it tells me that no partition was set. I was wandering if there is something obvious missing or if you have suggestion on how to troubleshoot this.
thanks!

[wlandau]

In newer versions of future, the resources argument of future::future() is no longer supported (futureverse/future.batchtools#75). Instead, you can define an entire future plan for a target. Sketch:

# _targets.R file
library(targets)
library(future)
plan <- tweak(
  strategy = batchtools_slurm,
  template = "batchtools.slurm.tmpl",
  ncpus = 1,
  memory = 16000,
  walltime = 600,
  partition = "scu_cpu"
)
resources_all <- tar_resources(
  clustermq = tar_resources_clustermq(template = list(n_cores = 1, memory = 150000)),
  future = tar_resources_future(plan = plan)
)
tar_option_set(resources = resources_all)
list(
  tar_target(x, command(), resources = resources_all),
  ...,
)

[c1au6i0]

Thank you! I am using tweak and I am defining future = tar_resources_future(plan = plan)
it still does not get the options defined in resources but now I get a some what more informative error message:

Warning message:
-n warnf("Detected %d unknown future arguments: %s", length(unknown),  :
 Detected 6 unknown future arguments: ‘ncpus’, ‘memory’, ‘walltime’, ‘partition’, ‘job.name’, ‘log.file’
Error: callr subprocess failed: Fatal error occurred: 101. Command 'sbatch' produced exit code 255. Output: 'sbatch: error: Invalid --time specification'
Visit https://books.ropensci.org/targets/debugging.html for debugging advice.
Run `rlang::last_error()` to see where the error occurred.

Using following version of packages:
[1] future.batchtools_0.10.0 future_1.23.0 targets_0.9.0

[wlandau]

Ah, you may have to write this:

tweak(
  strategy = batchtools_slurm,
  template = "batchtools.slurm.tmpl",
  resources = list(
    ncpus = 1,
    memory = 16000,
    walltime = 600,
    partition = "scu_cpu"
  )
)

Otherwise, I am betting this will reproduce the error with just future (no targets):

library(future)
library(future.batchtools)
plan(
  strategy = batchtools_slurm,
  template = "batchtools.slurm.tmpl",
  resources = list(
    ncpus = 1,
    memory = 16000,
    walltime = 600,
    partition = "scu_cpu"
  )
)
f <- future(1 + 1)
while (!resolved(f))
value(f)

If you still get errors at that point (reproducible with just future) then it may be time to follow up at futureverse/future.batchtools#75 or https://github.com/mllg/batchtools/issues.

[c1au6i0]

Yeap, that was it. Many thanks @wlandau !