EXPECTED FAIL STATUS
Pattern summary
Comparing to an Expected Fail outcome rather than the Expected Result enables automated failure analysis when there is a minor bug affecting all tests.
Category
Execution
Context
This pattern is useful when there is a minor bug that isn't going to fixed anytime soon, which is affecting a large number of automated tests. Every test is failing because of the same unimportant bug. But every failing automated test has to be investigated - this makes the failure analysis time become excessive, which is not good - wasteful. Sometimes people just don't bother to run the tests - then you have lost the value of the automated tests. Setting up an additional Test Status (not just Pass or Fail) gives additional benefit from automation and overcomes the problem.
Description
In addition to the "normal" test status of "Pass" or "Fail", two additional test statuses are needed (and even more can also be useful).
Pass: the test actual results match the Expected Results
Fail: the test actual results do not match the Expected Results
Expected Fail: the test results match the "Expected Fail Results" which include the minor bug as part of this test result.
Unknown: the test results do not match the Expected Fail Results.
Implementation
Using more than two test statuses needs to be set up ideally from the beginning of an automation effort, so that every test has the option of using additional statuses.
You need to specify which tests are to be compared to Expected Fail Results, and the Expected Fail Results need to be set up fro each of the tests that will use it. This may just be an (automated) update to a set of Expected Results.
When a test is run that does not match the Expected Fail Result, there are two possibilities - it may be that this minor bug has now been fixed, so running the test immediately against the Expected Result would result in a Pass. Or it could be that there is a new bug that the automated test has now found in addition to the known bug - in this case the test status is Unknown, and needs to be looked at.
Expected Fail is not the only useful additional test status - you could have a test status for any of the following (or others):
- tests blocked due to environment problem (e.g. network down, timeouts)
- set-up problem (e.g. files missing)
- test needs to be changed but hasn't been updated yet
Potential problems
It does take time to set up the facility to use more than two test statuses, but the benefits are also significant.
Issues addressed by this pattern
Experiences
If you have used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.