TAP flavour for subtests

[eliasvasylenko]

Specification work on TAP stalled before settling on subtest semantics.

Looking at the JS community (and to some extent e.g. the Perl community) it looks like people who want to pin subtest semantics onto the side of TAP 13 use some variation on the four-space-indent encoding proposed in the above issue.

It has a lot going for it;

• Already seen some adoption.
• Existing consumers should at least be forwards compatible with it, if they don't support it directly.
  • No false negatives.
  • No false positives, given the requirement that a parent test MUST fail if any enclosed subtests fail.
• Supports streaming.
• Easy to read for humans; we intuitively understand indentation as an encoding of structure.
• Easy to read for computers and composes well; a subtest is essentially itself a valid TAP stream, with just a simple line-by-line transformation to embed into/extract from an enclosing stream.

But it has some outstanding objections;

• indentation is also used to help delineate YAML sections and this can confuse some parsers.
• Concurrency. It is an inherently serial format, meaning subtests either have to be run one at a time, or their results have to be buffered so they can be pushed out all at once.

[eliasvasylenko]

Regarding the concurrency, i do think there is a possible solution that supports streaming.

Given that TestAnything/Specification#14, would break forwards compatibility, I would instead challange the prevailing assumption in the #2 thread that we need to associate a subtest line with the number of its enclosing test point. Instead we can associate it with the description. This is something like that was touched on here TestAnything/Specification#15, but again no consensus was reached.

So to illustrate, taking the following results:

    1..2
    ok 1 - a subtest
    ok 2 - a second subetest
ok 1 - some test
        ok 1 - sub-sub-test
        1..1
    ok 1 - a subtest
    ok 2 - subtest number two
    1..2
ok 2 - another test

If the producer wants to execute tests and subtests concurrently, it can prefix them with tags instead of just spaces:

[some test]  1..2
[some test]  ok 1 - a subtest
[another test]  [a subtest]  ok 1 - sub-sub-test
[another test]  [a subtest]  1..1
[another test]  ok 1 - a subtest
[another test]  ok 2 - subtest number two
[another test]  1..2
ok 1 - another test
[some test]  ok 2 - a second subetest
ok 2 - some test

There's a lot of repetition, but it may be more readable that the alternatives.

• Using human-readable text instead of a number makes associated lines easier to differentiate a glance as they are more visually distinct that a sea of numbers.
• Formatting the prefix as something like [test description] or test description| creates a sort of margin, so it's not too hard to refocus attention to the embedded TAP for a subtest.
• For consumers it only requires a generalisation of the same machinery behind reading indented subtests. Incremental extra complexity, one more layer of abstraction, rather than completely new and different semantics.