Difference between revisions of "FALSE PASS"
Jump to navigation
Jump to search
m (Topic titles in capital letters) |
|||
| Line 1: | Line 1: | ||
<div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Issues]] / Back to [[Test Automation Issues]]</span> | <div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Issues]] / Back to [[Test Automation Issues]]</span> | ||
| − | =<span style="font-family: Arial; font-size: 16px">Issue summary</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Issue summary'''</span>= |
<span style="font-family: Arial; font-size: 16px">The tests pass even if the SUT actually reacts erroneously</span> | <span style="font-family: Arial; font-size: 16px">The tests pass even if the SUT actually reacts erroneously</span> | ||
| − | =<span style="font-size: 16px">Category</span>= | + | =<span style="font-size: 16px">'''Category'''</span>= |
<span style="font-size: 16px">Execution </span> | <span style="font-size: 16px">Execution </span> | ||
| − | =<span style="font-size: 16px">Examples</span>= | + | =<span style="font-size: 16px">'''Examples'''</span>= |
# <span style="font-family: Arial; font-size: 16px">The automated tests drive the SUT, but don't compare the results with expected results so that they always pass</span> | # <span style="font-family: Arial; font-size: 16px">The automated tests drive the SUT, but don't compare the results with expected results so that they always pass</span> | ||
| Line 10: | Line 10: | ||
# <span style="font-family: Arial; font-size: 16px">Test automation has been developed with an erroroneus SUT and the results have been accepted as expected results without first checking that they are correct</span> | # <span style="font-family: Arial; font-size: 16px">Test automation has been developed with an erroroneus SUT and the results have been accepted as expected results without first checking that they are correct</span> | ||
# <span style="font-family: Arial; font-size: 16px">The tests aren't picking up bugs even though they are comparing the results for the right things and should find them.</span> | # <span style="font-family: Arial; font-size: 16px">The tests aren't picking up bugs even though they are comparing the results for the right things and should find them.</span> | ||
| − | =<span style="font-family: Arial; font-size: 16px">Questions</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Questions'''</span>= |
<span style="font-family: Arial; font-size: 16px">How are the expected results collected?</span><br /> <span style="font-family: Arial; font-size: 16px">Have the tests been sufficiently tested?</span> | <span style="font-family: Arial; font-size: 16px">How are the expected results collected?</span><br /> <span style="font-family: Arial; font-size: 16px">Have the tests been sufficiently tested?</span> | ||
| − | =<span style="font-family: Arial; font-size: 16px">Resolving Patterns</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Resolving Patterns'''</span>= |
<span style="font-size: 16px">Most recommended:</span><br /> | <span style="font-size: 16px">Most recommended:</span><br /> | ||
| Line 21: | Line 21: | ||
<br /> <span style="font-family: Arial; font-size: 16px">Other useful Patterns:</span><br /> | <br /> <span style="font-family: Arial; font-size: 16px">Other useful Patterns:</span><br /> | ||
| − | * | + | * <span style="font-size: 16px">[[MAINTAIN THE TESTWARE]]: use this pattern to keep your automation workable.</span> |
* <span style="background-color: transparent; color: #0066cc; display: block; font-family: arial,helvetica,sans-serif; font-size: 16px; text-align: left">[[SENSITIVE COMPARE]]<span style="background-color: transparent; color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; text-decoration: none">: If the tests are missing bugs because they are comparing too narrow a set of results (e.g. only the changed fields), the use of sensitive comparison can find unexpected results, where the test (correctly) fails.</span></span> | * <span style="background-color: transparent; color: #0066cc; display: block; font-family: arial,helvetica,sans-serif; font-size: 16px; text-align: left">[[SENSITIVE COMPARE]]<span style="background-color: transparent; color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; text-decoration: none">: If the tests are missing bugs because they are comparing too narrow a set of results (e.g. only the changed fields), the use of sensitive comparison can find unexpected results, where the test (correctly) fails.</span></span> | ||
<br /> <span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Issues]] / Back to [[Test Automation Issues]]</span></div> | <br /> <span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Issues]] / Back to [[Test Automation Issues]]</span></div> | ||
Latest revision as of 14:50, 28 April 2018
.................................................................................................................Main Page / Back to Execution Issues / Back to Test Automation Issues
.................................................................................................................Main Page / Back to Execution Issues / Back to Test Automation Issues
Issue summary
The tests pass even if the SUT actually reacts erroneously
Category
Execution
Examples
- The automated tests drive the SUT, but don't compare the results with expected results so that they always pass
- The new results are erroneously written over the expected results so that the compares always pass
- Test automation has been developed with an erroroneus SUT and the results have been accepted as expected results without first checking that they are correct
- The tests aren't picking up bugs even though they are comparing the results for the right things and should find them.
Questions
How are the expected results collected?
Have the tests been sufficiently tested?
Resolving Patterns
Most recommended:
- COMPARISON DESIGN: design the comparison of test results to be as efficient as possible, balancing Dynamic and Post-Execution Comparison, and using a mixture of Sensitive and Robust/ Specific comparisons.
- RIGHT INTERACTION LEVEL: make sure that your tests interact with the SUT at the most effective Level
- TEST THE TESTS: This is an important pattern to make sure that your tests are doing what you think they are doing.
- VERIFY-ACT-VERIFY: use this pattern if you cannot apply FRESH SETUP.
Other useful Patterns:
- MAINTAIN THE TESTWARE: use this pattern to keep your automation workable.
- SENSITIVE COMPARE: If the tests are missing bugs because they are comparing too narrow a set of results (e.g. only the changed fields), the use of sensitive comparison can find unexpected results, where the test (correctly) fails.
.................................................................................................................Main Page / Back to Execution Issues / Back to Test Automation Issues