Clarifying resource priority

[stuvet]

Hi,

Having learned by playing, I ended up with a messy codebase which had multiple levels of resource definitions. I've cleaned up my code but I was left wondering if the layered resource definitions had led to some of the unexpected behaviour I'd noticed, particularly with apparently premature ending of walltimes even on non-preemptible machines.

Example of 'messy' resource definitions:

library(targets)

slurm <- future::tweak(
                 batchtools_slurm, 
                 template = 'batchtools.slurm.tmpl', 
                 workers = 30L,
                 resources = list(
                   walltime = 3600,
                   ncpus = 4,
                   ntasks = 1,
                   memory = 1750,
                   partition = 'default'
                )
             )

future::plan(slurm)

tar_option_set(
         resources = tar_resources(
                aws = tar_resources_aws(bucket = "my_bucket"),
                future = tar_resources_future(
                   plan = future::plan(slurm),
                   resources = list(
                     walltime = 36000,
                     partition = 'pm2',
                     memory = 14400,
                     workers = 10L,
                     ncpus = 1,
                     ntasks = 1
                 )
            )
       )
)



list(
   tar_target(
        my_target,
        Sys.sleep(60),
        deployment = 'worker',
        resources =   tar_resources(
                aws = tar_resources_aws(bucket = "my_bucket"),
                future = tar_resources_future(
                   plan = future::plan(slurm),
                   resources = list(
                     walltime = 72000,
                     partition = 'ph32',
                     memory = 1750,
                     workers = 30L,
                     ncpus = 4,
                     ntasks = 1
                 )
             )
       )
  ) 
)

As part of the scourge I removed the resources & workers keys in the future::tweak statement & am testing at the moment, but could you please clarify (& perhaps in the manual) how targets prioritises layered resource definitions like these? Apologies if I've missed it.

Thanks.

[wlandau]

targets currently ignores the resources you supply to future::plan() and/or future::tweak(). (Maybe it should not do that.) The resources supplied to tar_target() completely supersede those supplied to tar_option_set(). So in your case, my_target should use a walltime of 72000, and any target with no resources specified at all will use a walltime of 36000. Also, in targets, resource specification is all or nothing. In other words, if you specify some resources in tar_target() but omit walltime, then the walltime of that target will default to whatever is in the batchtools template file, not tar_option_get("resources").

[wlandau]

Just added some clarifying comments to the Rd help files.

[stuvet]

Perfect. Thanks. That's pretty much what I assumed (though I was on the fence about fully-replaced vs partially-updated specifications). Turns out that doesn't explain the occasional odd walltime, though I've had to make some edits to my batchtools' namespace even to come close to stability on my slurm infrastructure (e.g. this) so I can't yet expect perfection.

Edit: odd walltimes could be slurm-level preemption rather than GCP-level preemption.

[wlandau]

Glad that helped.

Related: unfortunately, due to conflicts with the previous deprecated resource system (#504), targets cannot safely accept resources defined in future::plan() or future::tweak(). Maybe we can add that when unstructured resources are totally defunct later on.

[stuvet]

Worth knowing for others, but not an issue for me even with my workarounds. Thanks for clarification though.

[wresch]

I am a bit confused. Starting out with a very similar setup:

versions

 other attached packages:                                           
 [1] targets_0.7.0            future.batchtools_0.10.0 future_1.22.1

_targets.R

library(targets)
library(future.batchtools)

slurm <- future::tweak(batchtools_slurm, template = 'slurm')
future::plan(slurm)

list(
   tar_target(
        my_target,
        Sys.sleep(60),
        deployment = 'worker',
        resources =   tar_resources(
                future = tar_resources_future(
                   plan = future::plan(slurm),
                   resources = list(
                     walltime = 20,
                     partition = 'quick',
                     memory = 1750,
                     ncpus = 4
                 )
             )
       )
  )
)

batchtools.slurm.tmpl

#!/bin/bash
## Job Resource Interface Definition
##
## ncpus [integer(1)]:        Number of required cpus per task,
##                            Set larger than 1 if you want to further parallelize
##                            with multicore/parallel within each task.
## walltime [integer(1)]:     Walltime for this job, in minutes.
## memory   [integer(1)]:     Memory in megabytes.
##                            Must be at least 100 (when I tried lower values my
##                            jobs did not start at all).
##
## optional: partition, gres
##
## Default resources can be set in your .batchtools.conf.R by defining the variable
## 'default.resources' as a named list.
<%
# relative paths are not handled well by Slurm
log.file = fs::path_expand(log.file)
-%>
#SBATCH --job-name=<%= job.name %>
#SBATCH --output=<%= log.file %>
#SBATCH --error=<%= log.file %>
#SBATCH --time=<%= resources$walltime %>
#SBATCH --cpus-per-task=<%= resources$ncpus %>
#SBATCH --mem=<%= resources$memory %>
<%= if (!is.null(resources$partition)) sprintf("#SBATCH --partition=%s", resources$partition) -%>
<%= if (!is.null(resources$gres)) sprintf("#SBATCH --gres=%s", resources$gres) -%>
<%= if (array.jobs) sprintf("#SBATCH --array=1-%i", nrow(jobs)) else "" %>

## Initialize work environment like
## source /etc/profile
## module add ...
module load R/4.1

## Run R:
## we merge R output with stdout from SLURM, which gets then logged via --output option
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

Running tar_make_future(workers=1 however fails with

Error : Fatal error occurred: 101. Command 'sbatch' produced exit code 1. Output: 'sbatch: error: Invalid --time specification'

And looking at the processed script in the .future directory, I see that apparently the template got handed an empty resources list. Am I missing something?

[wlandau]

You can temporarily solve the issue by downgrading to future version 1.21.0. Worked on my end.

remotes::install_version("future", version = "1.21.0")

[wresch]

Indeed - in that case the resources are not passed along to the template script. I wasn't aware that it's supposed to be possible to pass resources to future directly. Is that documented somewhere? I'm fairly new to future

[wresch]

Sorry I missed your posts above before replying. Thank you very much - that clarifies things

[wresch]

The downgrade to future 1.21.0 worked and now resources specified in tar_target work as expected. Thank you very much!

[wlandau]

Please have a look at Henrik's response to futureverse/future.batchtools#75. In subsequent versions of future, it looks like temporary future plans are the way to accomplish what you are trying to do. I updated https://books.ropensci.org/targets/hpc.html#future-configuration to explain how.