[help] Targets not cleaning up /tmp/Rmp on SLURM workers & mistakenly detecting success

[stuvet]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

In trying to produce a reprex for #1214 on SLURM I have noticed two problems:

1. /tmp/Rtmp* files aren't cleaned up between tasks - large files can accumulate & fill /tmp
   • Could well be up to the user, but I had not anticipated this behaviour.
   • In hindsight this behaviour may be related to the next point, since some tasks were failing, but targets was detecting them as having completed successfully.
2. targets mistakenly detects a target writing to a full /tmp as having successfully completed while slurm sacct reports that it failed, and the same job run via crew.cluster correctly reports the failure (shown below, apologies that system didn't allow a true reprex):

I should say that I'm new to the crew.cluster implementation of targets as I've been reluctant to upgrade from a working future.batchtools setup. Sorry if I've made an obvious error, and please let me know if I can help any further with this one.

Correct Behaviour via crew.cluster - Failure

library(crew.cluster)

storageDir <- NULL # Leave null for tempdir(), but beware this is probably a tmpfs - may trigger RAM errors?

file_chunk_mb <- 2^10
file_chunks_n <- 40L # Number of chunks to create the whole file. Must be positive integer.

slurm <-  crew_controller_slurm(
  workers = 2L,
  seconds_wall = 3600,
  seconds_launch = 90,
  seconds_idle = 120,
  # slurm_memory_gigabytes_per_cpu = 1.8,
  slurm_cpus_per_task = 1L,
  tasks_max = 1L,
  slurm_partition = 'ph2',
  script_lines = c(
    "export MODULEPATH=/etc/modulefiles:/usr/share/modulefiles:/apps/modulefiles",
    "source /etc/profile.d/modules.sh",
    "source ~/.bashrc",
    "module load R/4.2.0"
  )
)

slurm$start()

slurm$push('test', {
  filename  <- tempfile(tmpdir = if (!is.character(storageDir)) tempdir() else storageDir)

  system(sprintf('dd if=/dev/zero of=%s bs=1M count=%s oflag=append', filename, file_chunk_mb * file_chunks_n))

  return('Succeeded!')
})

slurm$wait()

slurm$pop()

dd: error writing ‘/tmp/Rtmp9R4Sxc/filec992808a15b’: No space left on device
30839+0 records in
30838+0 records out
32336838656 bytes (32 GB) copied, 7.20995 s, 4.5 GB/s
Error in mirai::mirai(.expr = expr_crew_eval, name = name, command = command,  : 
  no function to return from, jumping to top level

Incorrect Behaviour via targets - Success

library(targets)
library(crew.cluster)

storageDir <- NULL # Leave null for tempdir(), but beware this is probably a tmpfs - may trigger RAM errors?

file_chunk_mb <- 2^10
file_chunks_n <- 40L # Number of chunks to create the whole file. Must be positive integer.

slurm <-  crew_controller_slurm(
  workers = 2L,
  seconds_wall = 3600,
  seconds_launch = 90,
  seconds_idle = 120,
  # slurm_memory_gigabytes_per_cpu = 1.8,
  slurm_cpus_per_task = 1L,
  tasks_max = 1L,
  slurm_partition = 'ph2',
  script_lines = c(
    "export MODULEPATH=/etc/modulefiles:/usr/share/modulefiles:/apps/modulefiles",
    "source /etc/profile.d/modules.sh",
    "source ~/.bashrc",
    "module load R/4.2.0"
  )
)

tar_option_set(
  deployment = 'worker',
  storage = 'worker',
  controller = slurm,
  error = 'stop'
)

list(
  tar_target(
    should_error_if_big,
    {
      filename  <- tempfile(tmpdir = if (!is.character(storageDir)) tempdir() else storageDir)

      system(sprintf('dd if=/dev/zero of=%s bs=1M count=%s oflag=append', filename, file_chunk_mb * file_chunks_n))

      return('Succeeded!')
    }
  )
)

▶ dispatched target should_error_if_big
● completed target should_error_if_big [0.28 seconds]
▶ completed pipeline [43.462 seconds]

tar_read('should_error_if_big')
[1] "Succeeded!"

Slurm Job Information

sacct -j 2296941 ## That's the right job

       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
2296941      crew-4302+        ph2       anon          1     FAILED     15:0
2296941.bat+      batch                  anon          1     FAILED     15:0

scontrol show job 2296941

JobId=2296941 JobName=crew-430262c045eb56b8a42b2b16-1-e29dfbfe817a155555fb1db2
   UserId=anon GroupId=anon(xxxx) MCS_label=N/A
   Priority=4294261858 Nice=0 Account=define QOS=normal
   JobState=FAILED Reason=NonZeroExitCode Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=143:0
   RunTime=00:00:02 TimeLimit=1-00:00:00 TimeMin=N/A
   SubmitTime=2024-01-25T11:04:04 EligibleTime=2024-01-25T11:04:04
   AccrueTime=2024-01-25T11:04:04
   StartTime=2024-01-25T11:04:04 EndTime=2024-01-25T11:04:06 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-01-25T11:04:04
   Partition=ph2 AllocNode:Sid=slurm-cluster-controller:8403
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=slurm-cluster-compute-h2-0
   BatchHost=slurm-cluster-compute-h2-0
   NumNodes=1 NumCPUs=1 NumTasks=0 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=1,mem=1155M,node=1,billing=1
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryCPU=1155M MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/tmp/Rtmps293Tx/2d460e334abe7365e18cdb58-430262c045eb56b8a42b2b16-1.sh
   WorkDir=/home/stuart/projects/targets_test
   StdErr=/dev/null
   StdIn=/dev/null
   StdOut=/dev/null
   Power=
   NtasksPerTRES:0

Session Info

R version 4.2.0 (2022-04-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /apps/R/4.2.0/lib64/R/lib/libRblas.so
LAPACK: /apps/R/4.2.0/lib64/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8      
 [2] LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8   
 [6] LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8      
 [8] LC_NAME=C                 
 [9] LC_ADDRESS=C              
[10] LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8
[12] LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices datasets  utils    
[6] methods   base     

other attached packages:
[1] crew.cluster_0.2.0 targets_1.4.1.9002

loaded via a namespace (and not attached):
 [1] igraph_1.6.0       rstudioapi_0.15.0 
 [3] knitr_1.45         xml2_1.3.6        
 [5] magrittr_2.0.3     tidyselect_1.2.0  
 [7] nanonext_0.12.0    secretbase_0.1.0  
 [9] R6_2.5.1           rlang_1.1.3       
[11] fansi_1.0.6        tools_4.2.0       
[13] parallel_4.2.0     data.table_1.14.10
[15] xfun_0.41          utf8_1.2.4        
[17] cli_3.6.2          withr_3.0.0       
[19] yaml_2.3.8         digest_0.6.34     
[21] base64url_1.4      tibble_3.2.1      
[23] lifecycle_1.0.4    crew_0.8.0        
[25] processx_3.8.3     mirai_0.12.0      
[27] callr_3.7.3        vctrs_0.6.5       
[29] ps_1.7.6           codetools_0.2-18  
[31] glue_1.7.0         getip_0.1-4       
[33] compiler_4.2.0     pillar_1.9.0      
[35] backports_1.4.1    renv_0.15.5       
[37] pkgconfig_2.0.3   

[wlandau]

1. targets does not control cleanup in /tmp in general, it depends on what exactly is in those /tmp files and who/what exactly is writing them. A bit more context would help.
2. Out-of-memory errors in general are hard to catch. (I am surprised crew caught that one.) The inconsistency is still a mystery, but if you set up log files using crew_controller_slurm(), then you may have more of a chance of detecting them.

[wlandau]

Testing out-of-memory or out-of-storage errors are pretty invasive, and I hesitate to do something that will crash my system or the system of someone else. If you can generate your same findings with a less invasive test, I can troubleshoot on my end.

[stuvet]

Cleanup

This was an attempt at writing a test/reprex for a situation where high storage usage on a SLURM worker propagated an 'OOM' RAM-style error.

The files written by the dd call are just full of zeros to give a required file size, & written to thetempdir() for the session.

New tasks submitted to those workers with residual /tmp/Rtmp*/* files had fewer & fewer resources, with new tasks eventually hitting a 'could not allocate vector of size 100MB' error (RAM was being sucked up by tmpfs I think?).

Perhaps the lack of cleanup was because R exited with an error code, or because of the system call, but it may cause issues when running in a targets environment with reusable workers - I could catch the error in R, & provide cleanup on error, but then I presume targets would see a successful exit?

Personally I'd be happy for the behaviour to be outlined in the manual, but perhaps you can think of an alternative option to clean up the tempdir() on failure?

Exit code

SLURM caught it as a failure. I haven't looked into how crew.cluster & targets monitor completion, but maybe there's some error code that's not being interpreted correctly? I'll take a look at the logic tomorrow & investigate a bit more if I can.

[stuvet]

Testing out-of-memory or out-of-storage errors are pretty invasive, and I hesitate to do something that will crash my system or the system of someone else. If you can generate your same findings with a less invasive test, I can troubleshoot on my end.

Completely understand. I was deliberately taking the hit at my end as I can use a disposable environment. I'll refine it to touch a temporary file then exit with an error.

[wlandau]

The files written by the dd call are just full of zeros to give a required file size, & written to thetempdir() for the session.

New tasks submitted to those workers with residual /tmp/Rtmp*/* files had fewer & fewer resources, with new tasks eventually hitting a 'could not allocate vector of size 100MB' error (RAM was being sucked up by tmpfs I think?).

If user-level code in the tasks is writing those files in /tmp, then targets does not know how to clean them up, and the tasks would need to do that.

Perhaps the lack of cleanup was because R exited with an error code, or because of the system call, but it may cause issues when running in a targets environment with reusable workers - I could catch the error in R, & provide cleanup on error, but then I presume targets would see a successful exit?

Both crew and targets catch errors that R can catch and which do not crash R. Stuff that happens outside R would need to be reported by the platform somehow.

Personally I'd be happy for the behaviour to be outlined in the manual, but perhaps you can think of an alternative option to clean up the tempdir() on failure?

Manually writing to tempdir() is a specific case, and I am not sure it would fit with targets to anticipate this applying to everybody. Likewise for the manual: it is a compact set of chapters for general usage, and the list of possible failure modes would be too large to fit there.

[stuvet]

No problem. Thanks for your time. I can see how to handle these cases, but I must admit it was a little unexpected for me as the user (& I'm not new to targets), since we're typically relying on targets to delegate to workers via crew.cluster, & I hadn't expected each target not to have the resources id requested.

This will be an issue for people running big pipelines where some tasks are expected to hit wall time & large files are written to tmp - cmdstanr & stantargets is one use-case that springs to mind.

Wrapping each target in a cleanup function is pretty easy, unless it's being abstracted in stantargets or the file is written by a package function.