Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify how consumers should handle parse errors #4

Open
jonathanKingston opened this issue Jan 6, 2015 · 7 comments
Open

Specify how consumers should handle parse errors #4

jonathanKingston opened this issue Jan 6, 2015 · 7 comments

Comments

@jonathanKingston
Copy link
Member

With a new version of TAP brings the ability to make stricter the behaviour around parsing and consuming TAP.

I think that (quick notes please feed back):

  • Implementations should fail hard and fast

    • We may likely want to specify a fatal error type behaviour which consumers can choose to do
    • Failing hard should be the default behaviour
    • Implementations may however choose to ignore fatal errors but they still should be reported with the same importance as other test cases
    • Fatal errors should have set meanings codes which are human readable
  • Implementations should only support the version they were designed for, when encountering a new version of TAP, the test should fail

  • Reporters should also have a set of fatal errors that can be appended to their output which means something went fatally wrong with the tests but perhaps not within the tests themselves (Lets say a network error on the socket)

    • This could be something as simple as:
     TAP version 14
     1..10
     ok 1 - reason
     not ok fatal
    

    and/or interpretations might be expected to provide an out message at the end to confirm all was ok

     TAP version 14
     1..1
     ok 1 - reason
     ok complete
    

    These could be treated as less fatal than a bail out - however we could specify more around that instead.

@jonathanKingston
Copy link
Member Author

Some of these comments likely clash with some of the more ambiguous comments in the: http://testanything.org/philosophy.html

This is not very helpful as I don't find any of the forwards compatible notion of TAP easy to follow as the format is so open. If this philosophy is to be followed then we need to specify further more on how libraries should behave around the edge cases caused.

As previously mentioned with other producers, this philosophy perhaps isn't ever followed anyway despite its good intentions.

@jonathanKingston jonathanKingston mentioned this issue Jan 14, 2015
@Leont
Copy link

Leont commented Jan 14, 2015

Implementations should fail hard and fast

I suspect pragmas (#6) could be helpful here.

Implementations should only support the version they were designed for, when encountering a new version of TAP, the test should fail

Backwards compatibility is a must-have, but forwards compatibility has always been a strength. I'm not sure there's really a to disallow that, but I'm not sure of the reverse either.

Reporters should also have a set of fatal errors that can be appended to their output which means something went fatally wrong with the tests but perhaps not within the tests themselves (Lets say a network error on the socket)

I'm not sure I get your point here. Why would that be useful? Reusing ok for that seems highly confusing.

@jonathanKingston
Copy link
Member Author

Backwards compatibility is a must-have, but forwards compatibility has always been a strength. I'm not sure there's really a to disallow that, but I'm not sure of the reverse either.

A pragma could be used for this, I would prefer strict but as that has some meaning already we would be safer opting for another. Personally my paranoia would prefer the ability to lock to the version I am expecting to parse in case of breaking functionality in the future or a misbehaving producer that decides to invent its own 14.5 format perhaps.

My idea around producers having fatal errors was to allow for the YAML document if there was more structured data that could be provided. Perhaps we can just have the following document after the bail out! and provide meaningful standard error codes for use in the YAML.
The ok complete was similar to the report produced by TAP/Y in that it would allow consumers to check that it received all its output.

The other thought I had would be that we could specify error conditions for consumers, such that:

 TAP version 14
 pragma +always-complete
 1..1
 ok 1 - reason

could be treated as if it read:

 TAP version 14
 1..2
 ok 1 - reason
 not ok fatal 2 - consumer did not encounter a complete statement using 'always-complete' pragma 

@jonathanKingston
Copy link
Member Author

Scrap that, lets provide meaningful error codes for tests only.

I think there is value to adding YAML after the bail out however, like:

...
ok 2 - reason
Bail out!
time: ...
test-running: 3
trace: ...

@Leont
Copy link

Leont commented Jan 14, 2015

The other thought I had would be that we could specify error conditions for consumers

I think I like that idea (though in this particular case it's only useful if the plan comes at the start).

@Leont
Copy link

Leont commented Jan 14, 2015

I think there is value to adding YAML after the bail out however, like:

Possibly it's useful to add it to the version or plan statement too. Or just a general purpose keyword for meta-information about the test run.

@jonathanKingston
Copy link
Member Author

The other thought I had would be that we could specify error conditions for consumers

I think I like that idea (though in this particular case it's only useful if the plan comes at the start).

Pretty sure the pragmas will need to be always be above the tests they relate to (I shall add this question to #6).

However my thinking around this is that because we can't really control how tests are output to whatever the reporting interface is being consumed by there is the ability in the specification to explain how the consumer should treat certain failure cases and I think considering them as other tests could be the way forward (Certainly it would help easier documentation around error handling).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants