The problem was that if one of the two processes (recentLTRs or olderLTRs) does not find any hits, that leads to a bunch of empty files being created. Those empty files then trip up other processes downstream and crash the whole pipeline. This should probably be solved using native nextflow control structures, but I worked around that by creating empty or dummy files wherever necessary.
Now at least the pipeline runs through, but I am not sure if or how these dummy files influence other programs in the pipeline.

I used the wrong comment characters in my earlier patch. These added dummy files are still necessary in the latest iteration of nf-repeatmasking since the pipeline does crash otherwise.