-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Specify how consumers should handle parse errors #4
Comments
Some of these comments likely clash with some of the more ambiguous comments in the: http://testanything.org/philosophy.html This is not very helpful as I don't find any of the forwards compatible notion of TAP easy to follow as the format is so open. If this philosophy is to be followed then we need to specify further more on how libraries should behave around the edge cases caused. As previously mentioned with other producers, this philosophy perhaps isn't ever followed anyway despite its good intentions. |
I suspect pragmas (#6) could be helpful here.
Backwards compatibility is a must-have, but forwards compatibility has always been a strength. I'm not sure there's really a to disallow that, but I'm not sure of the reverse either.
I'm not sure I get your point here. Why would that be useful? Reusing ok for that seems highly confusing. |
A pragma could be used for this, I would prefer My idea around producers having fatal errors was to allow for the YAML document if there was more structured data that could be provided. Perhaps we can just have the following document after the The other thought I had would be that we could specify error conditions for consumers, such that:
could be treated as if it read:
|
Scrap that, lets provide meaningful error codes for tests only. I think there is value to adding YAML after the bail out however, like:
|
I think I like that idea (though in this particular case it's only useful if the plan comes at the start). |
Possibly it's useful to add it to the version or plan statement too. Or just a general purpose keyword for meta-information about the test run. |
Pretty sure the pragmas will need to be always be above the tests they relate to (I shall add this question to #6). However my thinking around this is that because we can't really control how tests are output to whatever the reporting interface is being consumed by there is the ability in the specification to explain how the consumer should treat certain failure cases and I think considering them as other tests could be the way forward (Certainly it would help easier documentation around error handling). |
With a new version of TAP brings the ability to make stricter the behaviour around parsing and consuming TAP.
I think that (quick notes please feed back):
Implementations should fail hard and fast
Implementations should only support the version they were designed for, when encountering a new version of TAP, the test should fail
Reporters should also have a set of fatal errors that can be appended to their output which means something went fatally wrong with the tests but perhaps not within the tests themselves (Lets say a network error on the socket)
and/or interpretations might be expected to provide an out message at the end to confirm all was ok
These could be treated as less fatal than a bail out - however we could specify more around that instead.
The text was updated successfully, but these errors were encountered: