revdepcheck::revdep_check() stall on 'targets' when checking in parallel

[HenrikBengtsson]

Hi.

Problem

I run revdepcheck::revdep_check(num_workers = n) on my future package with n > 1 (i.e. check packages in parallel). Each time (=reproducible), I end up having to kill the checks on targets, because one of the two runs (towards "new" and "old" versions of future) stalls. The other one always completes. Looking at htop, it seems like this happens when R CMD check is running the unit tests of targets.

This happens only for targets and not for any of the other ~250 packages. I've observed this for a couple of years now, so it's nothing new.

Workaround

To solve this, I end up terminating revdepcheck::revdep_check(). Then I add back targets to check that by itself, and run with revdepcheck::revdep_check(num_workers = 1L). This will cause the "old" version of targets to be checked first, and then the "new" version of targets. This always works and completes.

Troubleshooting

I suspect that there is a conflict when checking "old" and "new" runs of targets concurrently. Note that regardless of num_workers = n value, the package checks may run in parallel as usual.

@wlandau, can you think of anything in the unit tests that might cause a race condition?

[HenrikBengtsson]

More information from the logs:

The "old" check finished, whereas the "new" timed out after two hours:

$ cat 00check.log
* using log directory ‘/c4/home/henrik/repositories/future/revdep/checks/targets/new/targets.Rcheck’
* using R version 4.1.2 (2021-11-01)
* using platform: x86_64-pc-linux-gnu (64-bit)
* using session charset: UTF-8
* using option ‘--no-manual’
* checking for file ‘targets/DESCRIPTION’ ... OK
* this is package ‘targets’ version ‘0.10.0’
...
* checking files in ‘vignettes’ ... OK
* checking examples ... OK
* checking for unstated dependencies in ‘tests’ ... OK
* checking tests ... [TIMEOUT HERE]

$ ls -lrt tests/
total 5
drwxr-xr-x 2 henrik cbc   6 Feb 19 00:20 performance
drwxr-xr-x 2 henrik cbc  22 Feb 19 00:20 interactive
drwxr-xr-x 2 henrik cbc  12 Feb 19 00:20 hpc
drwxr-xr-x 2 henrik cbc   8 Feb 19 00:20 aws
-rw-r--r-- 1 henrik cbc 106 Feb 19 00:20 testthat.R
-rw-r--r-- 1 henrik cbc 131 Feb 19 00:20 startup.Rs
-rw-r--r-- 1 henrik cbc 841 Feb 19 00:20 testthat.Rout
drwxr-xr-x 3 henrik cbc 187 Feb 19 00:20 testthat

$ cat tests/testthat.Rout 
R version 4.1.2 (2021-11-01) -- "Bird Hippie"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

...

> Sys.setenv(PROCESSX_NOTIFY_OLD_SIGCHLD = "true")
> library(testthat)
> library(targets)

Attaching package: 'targets'

The following object is masked from 'package:testthat':

    matches

> test_check("targets")

Is there a way to see how far testthat made it? I cannot find any output indicating "progress" other than the latter above.

[HenrikBengtsson]

Hi,

thanks for this. Unfortunately, It still stalls.

Here's what I did:

I check out commit 24db22f and ran R CMD build to get targets_0.10.0.9000.tar.gz. Then I launched two parallel package checks:

mkdir check_1 check_2
for kk in {1..2}; do R CMD check --output="check_${kk}" --as-cran targets_0.10.0.9000.tar.gz &> "check_${kk}.out" & done
#$ [1] 27235
#$ [2] 27236

One of them completes, and the other gets stuck at:

* checking tests ...
  Running ‘testthat.R’

UPDATE: I'm running with up-to-date dependencies, including clustermq 0.8.95.3.

[wlandau]

I reproduced the stalling you see, and removing clustermq seems to fix it. I do not know what the underlying issue is, but in 8f91e50 I added a new environment variable to suppress those tests. If I set TAR_SKIP_CLUSTERMQ to any value, I can now run two concurrent check workflows without any stalling.

[wlandau]

None of the clustermq tests use future, so the revdep check should be just as good for your purposes.

[wlandau]

Running 2 concurrent tar_make_clustermq() calls in 2 separate projects outside the testing workflow seems to work.

[HenrikBengtsson]

I could finally test this; I ran revdepcheck on the new targets 0.11.0 with TAR_SKIP_CLUSTERMQ=true set and can confirm that the targets old-new tests no longer stalls when run concurrently. Thxs.