View source for EXPECTED FAIL STATUS

<div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Patterns]]/ Back to [[Test Automation Patterns]]</span>
=<span style="font-size: 16px">'''Pattern summary'''</span>=
<span style="font-size: 16px">Comparing to an Expected Fail outcome rather than the Expected Result enables automated failure analysis when there is a minor bug affecting all tests.</span>
=<span style="font-size: 16px">'''Category'''</span>=
<span style="font-size: 16px">Execution</span>
=<span style="font-size: 16px">'''Context'''</span>=
<span style="font-size: 16px">This pattern is useful when there is a minor bug that isn't going to fixed anytime soon, which is affecting a large number of automated tests. Every test is failing because of the same unimportant bug. But every failing automated test has to be investigated - this makes the failure analysis time become excessive, which is not good - wasteful. Sometimes people just don't bother to run the tests - then you have lost the value of the automated tests. Setting up an additional Test Status (not just Pass or Fail) gives additional benefit from automation and overcomes the problem.</span>
=<span style="font-size: 16px">'''Description'''</span>=
<span style="font-size: 16px">In addition to the "normal" test status of "Pass" or "Fail", two additional test statuses are needed (and even more can also be useful).</span><br /> <span style="font-size: 16px">Pass: the test actual results match the Expected Results</span><br /> <span style="font-size: 16px">Fail: the test actual results do not match the Expected Results</span><br /> <span style="font-size: 16px">Expected Fail: the test results match the "Expected Fail Results" which include the minor bug as part of this test result.</span><br /> <span style="font-size: 16px">Unknown: the test results do not match the Expected Fail Results.</span>
=<span style="font-size: 16px">'''Implementation'''</span>=
<span style="font-size: 16px">Using more than two test statuses needs to be set up ideally from the beginning of an automation effort, so that every test has the option of using additional statuses. </span><br /> <br /> <span style="font-size: 16px">You need to specify which tests are to be compared to Expected Fail Results, and the Expected Fail Results need to be set up fro each of the tests that will use it. This may just be an (automated) update to a set of Expected Results.</span><br /> <span style="font-size: 16px">When a test is run that does not match the Expected Fail Result, there are two possibilities - it may be that this minor bug has now been fixed, so running the test immediately against the Expected Result would result in a Pass. Or it could be that there is a new bug that the automated test has now found in addition to the known bug - in this case the test status is Unknown, and needs to be looked at.</span><br /> <br /> <span style="font-size: 16px">Expected Fail is not the only useful additional test status - you could have a test status for any of the following (or others):</span><br /> <span style="font-size: 16px">- tests blocked due to environment problem (e.g. network down, timeouts)</span><br /> <span style="font-size: 16px">- set-up problem (e.g. files missing)</span><br /> <span style="font-size: 16px">- test needs to be changed but hasn't been updated yet</span>
=<span style="font-size: 16px">'''Potential problems'''</span>=
<span style="font-size: 16px">It does take time to set up the facility to use more than two test statuses, but the benefits are also significant.</span>
=<span style="font-size: 16px">'''Issues addressed by this pattern'''</span>=
<span style="font-size: 16px">[[INEFFICIENT FAILURE ANALYSIS]]</span>
=<span style="font-size: 16px">'''Experiences'''</span>=
<span style="font-size: 16px">If you have used this pattern, please add your name and a brief story of how you used this pattern: your context, what you did, and how well it worked - or how it didn't work!</span><br /> <span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Patterns]]/ Back to [[Test Automation Patterns]]</span></div>