Difference between revisions of "FALSE FAIL"
Jump to navigation
Jump to search
(Created page with "<div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.........................................................................................") |
m (Topic titles in capital letters) |
||
| Line 1: | Line 1: | ||
<div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Issues]] / Back to [[Test Automation Issues]]</span> | <div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Execution Issues]] / Back to [[Test Automation Issues]]</span> | ||
| − | =<span style="font-family: Arial; font-size: 16px">Issue summary</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Issue summary'''</span>= |
<span style="font-family: Arial; font-size: 16px">The tests fail not because of errors in the SUT, but because of errors in the test automation testware or environment issues</span> | <span style="font-family: Arial; font-size: 16px">The tests fail not because of errors in the SUT, but because of errors in the test automation testware or environment issues</span> | ||
| − | =<span style="font-size: 16px">Category</span>= | + | =<span style="font-size: 16px">'''Category'''</span>= |
<span style="font-size: 16px">Execution</span> | <span style="font-size: 16px">Execution</span> | ||
| − | =<span style="font-family: Arial; font-size: 16px">Examples</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Examples'''</span>= |
# <span style="font-family: Arial; font-size: 16px">Tests fail because a window pops up that was not considered when developing the automation</span> | # <span style="font-family: Arial; font-size: 16px">Tests fail because a window pops up that was not considered when developing the automation</span> | ||
| Line 11: | Line 11: | ||
# <span style="font-family: Arial; font-size: 16px">Tests fail because something else is running on the same machine</span> | # <span style="font-family: Arial; font-size: 16px">Tests fail because something else is running on the same machine</span> | ||
# <span style="font-family: Arial; font-size: 16px">Tests fail because the database has been corrupted or changed by another application</span> | # <span style="font-family: Arial; font-size: 16px">Tests fail because the database has been corrupted or changed by another application</span> | ||
| − | # <span style="font-family: Arial; font-size: 16px">Tests fail because you are using a [[SENSITIVE COMPARE ]]and so changes that have nothing to do with your test case affect the results</span> | + | # <span style="font-family: Arial; font-size: 16px">Tests fail because you are using a [[SENSITIVE COMPARE]] and so changes that have nothing to do with your test case affect the results</span> |
| − | =<span style="font-family: Arial; font-size: 16px">Questions</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Questions'''</span>= |
<span style="font-family: Arial; font-size: 16px">Are the initial conditions set correctly?</span><br /> <span style="font-family: Arial; font-size: 16px">Are there dependencies between the test cases?</span><br /> <span style="font-family: Arial; font-size: 16px">Have the tests been sufficiently tested?</span> | <span style="font-family: Arial; font-size: 16px">Are the initial conditions set correctly?</span><br /> <span style="font-family: Arial; font-size: 16px">Are there dependencies between the test cases?</span><br /> <span style="font-family: Arial; font-size: 16px">Have the tests been sufficiently tested?</span> | ||
| − | =<span style="font-family: Arial; font-size: 16px">Resolving Patterns</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Resolving Patterns'''</span>= |
<span style="font-family: Arial; font-size: 16px">Most recommended:</span><br /> | <span style="font-family: Arial; font-size: 16px">Most recommended:</span><br /> | ||
Revision as of 14:42, 28 April 2018
.................................................................................................................Main Page / Back to Execution Issues / Back to Test Automation Issues
.................................................................................................................Main Page / Back to Execution Issues Execution Issues / Back to Test Automation Issues
Issue summary
The tests fail not because of errors in the SUT, but because of errors in the test automation testware or environment issues
Category
Execution
Examples
- Tests fail because a window pops up that was not considered when developing the automation
- Tests fail because the initial conditions have been corrupted by another test
- A test always passes if it runs by itself but always fails when run in the test suite
- Tests fail because something else is running on the same machine
- Tests fail because the database has been corrupted or changed by another application
- Tests fail because you are using a SENSITIVE COMPARE and so changes that have nothing to do with your test case affect the results
Questions
Are the initial conditions set correctly?
Are there dependencies between the test cases?
Have the tests been sufficiently tested?
Resolving Patterns
Most recommended:
- COMPARISON DESIGN: design the comparison of test results to be as efficient as possible, balancing Dynamic and Post-Execution Comparison, and using a mixture of Sensitive and Robust/ Specific comparisons.
- DEDICATED RESOURCES: Use this pattern if you have issues similar to Examples 4. or 5.
- FRESH SETUP: This pattern guards against issues like Example 2.
- INDEPENDENT TEST CASES: apply this pattern to get rid of issues like Example 3.
- RIGHT INTERACTION LEVEL: make sure that your tests interact with the SUT at the most effective level
Other useful patterns:
- MAINTAIN THE TESTWARE: apply this pattern to make sure that your automation keeps working even is the SUT is being changed
- SHARE INFORMATION: this pattern helps you learn in time when development is planning big or small changes that affect automation
- SPECIFIC COMPARE: Expected results are specific to the test case so changes to objects not processed in the test case don't affect the test results. Use this pattern for issues like Example 6
- TEST THE TESTS: use this pattern always (even for quick fixes!)
.................................................................................................................Main Page / Back to Execution Issues Execution Issues / Back to Test Automation Issues