Difference between revisions of "EXPECTED FAIL STATUS"
(Created page with "<div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.........................................................................................") |
|||
| (3 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
<div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Patterns]]/ Back to [[Test Automation Patterns]]</span> | <div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Patterns]]/ Back to [[Test Automation Patterns]]</span> | ||
| − | =<span style="font-size: 16px">Pattern summary</span>= | + | =<span style="font-size: 16px">'''Pattern summary'''</span>= |
<span style="font-size: 16px">Comparing to an Expected Fail outcome rather than the Expected Result enables automated failure analysis when there is a minor bug affecting all tests.</span> | <span style="font-size: 16px">Comparing to an Expected Fail outcome rather than the Expected Result enables automated failure analysis when there is a minor bug affecting all tests.</span> | ||
| − | =<span style="font-size: 16px">Category</span>= | + | =<span style="font-size: 16px">'''Category'''</span>= |
<span style="font-size: 16px">Execution</span> | <span style="font-size: 16px">Execution</span> | ||
| − | =<span style="font-size: 16px">Context</span>= | + | =<span style="font-size: 16px">'''Context'''</span>= |
| − | <span style="font-size: 16px">This pattern is useful when there is a minor bug that isn't going to fixed anytime soon, which is affecting a large number of automated tests. Every test is failing because of the same unimportant bug. But every failing automated | + | <span style="font-size: 16px">This pattern is useful when there is a minor bug that isn't going to fixed anytime soon, which is affecting a large number of automated tests. Every test is failing because of the same unimportant bug. But every failing automated test has to be investigated - this makes the failure analysis time become excessive, which is not good - wasteful. Sometimes people just don't bother to run the tests - then you have lost the value of the automated tests. Setting up an additional Test Status (not just Pass or Fail) gives additional benefit from automation and overcomes the problem.</span> |
| − | =<span style="font-size: 16px">Description</span>= | + | =<span style="font-size: 16px">'''Description'''</span>= |
<span style="font-size: 16px">In addition to the "normal" test status of "Pass" or "Fail", two additional test statuses are needed (and even more can also be useful).</span><br /> <span style="font-size: 16px">Pass: the test actual results match the Expected Results</span><br /> <span style="font-size: 16px">Fail: the test actual results do not match the Expected Results</span><br /> <span style="font-size: 16px">Expected Fail: the test results match the "Expected Fail Results" which include the minor bug as part of this test result.</span><br /> <span style="font-size: 16px">Unknown: the test results do not match the Expected Fail Results.</span> | <span style="font-size: 16px">In addition to the "normal" test status of "Pass" or "Fail", two additional test statuses are needed (and even more can also be useful).</span><br /> <span style="font-size: 16px">Pass: the test actual results match the Expected Results</span><br /> <span style="font-size: 16px">Fail: the test actual results do not match the Expected Results</span><br /> <span style="font-size: 16px">Expected Fail: the test results match the "Expected Fail Results" which include the minor bug as part of this test result.</span><br /> <span style="font-size: 16px">Unknown: the test results do not match the Expected Fail Results.</span> | ||
| − | =<span style="font-size: 16px">Implementation</span>= | + | =<span style="font-size: 16px">'''Implementation'''</span>= |
<span style="font-size: 16px">Using more than two test statuses needs to be set up ideally from the beginning of an automation effort, so that every test has the option of using additional statuses. </span><br /> <br /> <span style="font-size: 16px">You need to specify which tests are to be compared to Expected Fail Results, and the Expected Fail Results need to be set up fro each of the tests that will use it. This may just be an (automated) update to a set of Expected Results.</span><br /> <span style="font-size: 16px">When a test is run that does not match the Expected Fail Result, there are two possibilities - it may be that this minor bug has now been fixed, so running the test immediately against the Expected Result would result in a Pass. Or it could be that there is a new bug that the automated test has now found in addition to the known bug - in this case the test status is Unknown, and needs to be looked at.</span><br /> <br /> <span style="font-size: 16px">Expected Fail is not the only useful additional test status - you could have a test status for any of the following (or others):</span><br /> <span style="font-size: 16px">- tests blocked due to environment problem (e.g. network down, timeouts)</span><br /> <span style="font-size: 16px">- set-up problem (e.g. files missing)</span><br /> <span style="font-size: 16px">- test needs to be changed but hasn't been updated yet</span> | <span style="font-size: 16px">Using more than two test statuses needs to be set up ideally from the beginning of an automation effort, so that every test has the option of using additional statuses. </span><br /> <br /> <span style="font-size: 16px">You need to specify which tests are to be compared to Expected Fail Results, and the Expected Fail Results need to be set up fro each of the tests that will use it. This may just be an (automated) update to a set of Expected Results.</span><br /> <span style="font-size: 16px">When a test is run that does not match the Expected Fail Result, there are two possibilities - it may be that this minor bug has now been fixed, so running the test immediately against the Expected Result would result in a Pass. Or it could be that there is a new bug that the automated test has now found in addition to the known bug - in this case the test status is Unknown, and needs to be looked at.</span><br /> <br /> <span style="font-size: 16px">Expected Fail is not the only useful additional test status - you could have a test status for any of the following (or others):</span><br /> <span style="font-size: 16px">- tests blocked due to environment problem (e.g. network down, timeouts)</span><br /> <span style="font-size: 16px">- set-up problem (e.g. files missing)</span><br /> <span style="font-size: 16px">- test needs to be changed but hasn't been updated yet</span> | ||
| − | =<span style="font-size: 16px">Potential problems</span>= | + | =<span style="font-size: 16px">'''Potential problems'''</span>= |
<span style="font-size: 16px">It does take time to set up the facility to use more than two test statuses, but the benefits are also significant.</span> | <span style="font-size: 16px">It does take time to set up the facility to use more than two test statuses, but the benefits are also significant.</span> | ||
| − | =<span style="font-size: 16px">Issues addressed by this pattern</span>= | + | =<span style="font-size: 16px">'''Issues addressed by this pattern'''</span>= |
<span style="font-size: 16px">[[INEFFICIENT FAILURE ANALYSIS]]</span> | <span style="font-size: 16px">[[INEFFICIENT FAILURE ANALYSIS]]</span> | ||
| − | =<span style="font-size: 16px">Experiences</span>= | + | =<span style="font-size: 16px">'''Experiences'''</span>= |
| − | <span style="font-size: 16px">If you have used this pattern, please | + | |
| + | <span style="font-size: 16px">If you have used this pattern and would like to contribute your experience to the wiki, please go to [[Feedback]] to submit your experience or comment.</span><br /> <br /> | ||
| + | |||
| + | <span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Patterns]]/ Back to [[Test Automation Patterns]]</span></div> | ||
Latest revision as of 16:03, 21 August 2018
Pattern summary
Comparing to an Expected Fail outcome rather than the Expected Result enables automated failure analysis when there is a minor bug affecting all tests.
Category
Execution
Context
This pattern is useful when there is a minor bug that isn't going to fixed anytime soon, which is affecting a large number of automated tests. Every test is failing because of the same unimportant bug. But every failing automated test has to be investigated - this makes the failure analysis time become excessive, which is not good - wasteful. Sometimes people just don't bother to run the tests - then you have lost the value of the automated tests. Setting up an additional Test Status (not just Pass or Fail) gives additional benefit from automation and overcomes the problem.
Description
In addition to the "normal" test status of "Pass" or "Fail", two additional test statuses are needed (and even more can also be useful).
Pass: the test actual results match the Expected Results
Fail: the test actual results do not match the Expected Results
Expected Fail: the test results match the "Expected Fail Results" which include the minor bug as part of this test result.
Unknown: the test results do not match the Expected Fail Results.
Implementation
Using more than two test statuses needs to be set up ideally from the beginning of an automation effort, so that every test has the option of using additional statuses.
You need to specify which tests are to be compared to Expected Fail Results, and the Expected Fail Results need to be set up fro each of the tests that will use it. This may just be an (automated) update to a set of Expected Results.
When a test is run that does not match the Expected Fail Result, there are two possibilities - it may be that this minor bug has now been fixed, so running the test immediately against the Expected Result would result in a Pass. Or it could be that there is a new bug that the automated test has now found in addition to the known bug - in this case the test status is Unknown, and needs to be looked at.
Expected Fail is not the only useful additional test status - you could have a test status for any of the following (or others):
- tests blocked due to environment problem (e.g. network down, timeouts)
- set-up problem (e.g. files missing)
- test needs to be changed but hasn't been updated yet
Potential problems
It does take time to set up the facility to use more than two test statuses, but the benefits are also significant.
Issues addressed by this pattern
Experiences
If you have used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.