prove --rules does not work as advertised

[teodesian]

prove --version TAP::Harness v3.38 and Perl v5.26.0

Steps to Reproduce:

Read this section in the prove docs about --rules:
https://metacpan.org/pod/prove#-rules
The most practical use is likely to specify that some tests are not "parallel-ready".
That's exactly what I'm trying to do here.

I have 3 tests, all which sleep five seconds and then issue a pass().

prove -j3 --rules='seq=a.t' --rules='par=b.t' --rules='par=c.t' a.t b.t c.t
a.t ... ok
b.t ... ok
c.t .. ok
All tests successful.
Files=3, Tests=3, 6 wallclock secs ( 0.07 usr 0.03 sys + 0.31 cusr 0.10 csys = 0.51 CPU)
Result: PASS

As we can see above, the tests took 6s to run, which suggests every one of them ran in parallel (It should have taken 11s (1s prove overhead, 5s seq run, 5s par run).

This made me suspect the matching for full strings without globs was broken. However, in the prove docs it states:

Any test which does not match a rule will be run in sequence at the end of the run.

This did not occur, ergo the seq rule for a.t should have matched. Furthermore, since they were all run in parallel either way, that explanation didn't hold water. I then suspected that the matching order may be covering up a broken match:

"First match wins". The first rule that matches a test will be the one that applies.

However, I found no indication in the TAP::Parser::Scheduler source suggesting that specifying the test's full name would not result in a match. Nor did I see evidence that the order of rule processing was mishandled. In fact, TAP::Parser::Scheduler::_set_rules appeared to correctly classify my input:

#Output from dumping $self->{scheduler} before the return() in App::Prove::Scheduler->new()
$VAR1 = [
[
[
[
bless( {
'description' => 'a.t',
'filename' => 'a.t'
}, 'TAP::Parser::Scheduler::Job' )
]
]
],
[
bless( {
'description' => 'b.t',
'filename' => 'b.t'
}, 'TAP::Parser::Scheduler::Job' )
],
[
bless( {
'filename' => 'c.t',
'description' => 'c.t'
}, 'TAP::Parser::Scheduler::Job' )
]
];

After my initial shock at the dubious usefulness of such a data structure, I realized that the hugely nested first entry signified that the first entry indeed had walked the seq => codepath in _rule_clause, effectively ruling out any possibility of a match failure.

After more analysis of both TAP::Parser::Scheduler and TAP::Harness, the reason for all this becomes quite clear. The helper subs of Tap::Harness::aggregate_tests, _aggregate_parallell and _aggregate_single are just calling $scheduler->get_job*() and iterating in the sensible manner for the respective runmode.

However, the get_job() and get_jobs() subs return no information whatsoever about whether the job should be run in parallel or in sequence, and the helpers do nothing but call those subs. As such, the --rules information is effectively ignored (aside from throttling down -j to 1 when nothing but seq rules are passed). Having no information to work on it simply works the same way it did before the --rules feature was added.

As far as I can tell the right course of action would be:

• Actually check which jobs are parallel and which are seq in _aggregate_tests, by iterating over $scheduler->{schedule}, which has the information we want.
• Then call _aggregate_parallel or _aggregate_single, depending on what that test "wants", and whether our -j > 1.

The hard part with the latter, of course, is that we'd basically have to dummy up new TAP::Parser::Schedulers to feed into the aggregate_ helpers with the jobs restricted to each schedule "batch". When I say batch, I mean something like the third rules example in Tap::Parser::Scheduler:

{
seq => [
{ seq => 't/startup/*.t' }, #batch 1
{ par => ['t/a/*.t','t/b/*.t','t/c/*.t'], } #batch 2
{ seq => 't/shutdown/*.t' }, #batch3 ...
],
},

The data structure formed inside Tap::Parser::Scheduler::_set_rules is nowhere near as nice as that (it's deeply nested arrays as shown above in dumper output). That should be straightforward to clean up, given hardly anything is actually consulting it in practice.