Stop On Fail

People want an option to have the test stop after the first failure. This is a good idea, but the Test::Builder (and Test::Builder 1.5) design has an issue which makes this difficult to implement reliably. It has to do with how test functions are written and failure diagnostics produced.

Note: This is trivial to do with Test::Stream

https://github.com/Test-More/Test-Stream/commit/8988eb480e2b55486f0cdfea2650dfbcff3f8d79

https://github.com/Test-More/Test-Stream/commit/dc1ea9d6f93b2b25bc01ea2e25bae4156164256c

The Architecture Problem

A simple version of is() looks like this:

sub is {
    my($have, $want, $name) = @_;

    # Get the Test::Builder singleton to do the heavy lifting.
    my $tb = Test::Builder->new;

    # Do the actual test, prints ok #, the line and file and so on.
    my $ok = $tb->ok( $have eq $want, $name );

    # Prints out the failure diagnostics.
    $tb->diag(<<DIAGNOSTIC);
  have: '$have'
  want: '$want'
DIAGNOSTIC

    return $ok;
}

Normally when this fails, you'll see something like:

not ok 1 - this and that
# Failed test at foo.t line 42.
#   have: 'this'
#   want: 'that'

If the program exits when ok() fails, the $tb->diag line is never reached. You'll see this:

not ok 1 - this and that
# Failed test at foo.t line 42.

You'll know what failed, but you won't know why. This drastically reduces the value of the failing test as a debugging tool. This is why "stop on fail" is so difficult.

Commonly Suggested Solutions

There have been a lot of solutions to solve this. A lot of them have problems. Maybe they can be fixed.

use Test::Most

Test::Most has die_on_fail and bail_on_fail. It has caveats, but it basically works.

Add special cases for is() and like() and is_deeply() and ...

That might work for Test::More, but Test modules on CPAN are also written with Test::Builder and they have the same design issue. We can't fix them all.

Stop after the next diag()

As heuristics go, this isn't too bad, but there's no guarantee of it. It's arguable whether this is acceptable. It will not be sufficient where the test script uses more than one diag() after the ok() in order to describe the problem.

Stop after the next ok.

What if that test fails, too?

Stop after the next passing ok.

This is worth discussing.

Stop (bail out) at the end of the test file.

This is doable. While the whole .t file still runs, at least the test suite stops.