Difference between revisions of "FLAKY TESTS"
Jump to navigation
Jump to search
(Created page with "<div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.........................................................................................") |
m (Topic titles in capital letters) |
||
| Line 1: | Line 1: | ||
| − | <div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">................................................................................................................. | + | <div id="content_view" class="wiki" style="display: block"><span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[ Execution Issues]] / Back to [[ Test Automation Issues]]</span> |
| − | =<span style="font-family: Arial; font-size: 16px">Issue summary</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Issue summary'''</span>= |
<span style="font-family: Arial; font-size: 16px">The automated tests seem to pass or fail randomly </span> | <span style="font-family: Arial; font-size: 16px">The automated tests seem to pass or fail randomly </span> | ||
| − | =<span style="font-size: 16px">Category</span>= | + | =<span style="font-size: 16px">'''Category'''</span>= |
<span style="font-size: 16px">Execution</span> | <span style="font-size: 16px">Execution</span> | ||
| − | =<span style="font-family: Arial; font-size: 16px">Examples</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Examples'''</span>= |
| − | |||
# <span style="font-family: Arial; font-size: 16px">Mostly the tests pass but once in a while they fail. If they are rerun they pass as usual</span> | # <span style="font-family: Arial; font-size: 16px">Mostly the tests pass but once in a while they fail. If they are rerun they pass as usual</span> | ||
# <span style="font-family: Arial; font-size: 16px">Automated tests fail because development was using your same database and changed its structure</span> | # <span style="font-family: Arial; font-size: 16px">Automated tests fail because development was using your same database and changed its structure</span> | ||
| Line 14: | Line 13: | ||
# <span style="font-family: Arial; font-size: 16px">A previous test failed and left the system "dirty" so that all successive tests also failed</span> | # <span style="font-family: Arial; font-size: 16px">A previous test failed and left the system "dirty" so that all successive tests also failed</span> | ||
# <span style="font-family: Arial; font-size: 16px">Automated tests assume that something in the test environment is immutable, i.e. will not change, because the setup is assumed to be under the testers' control and deviations can be corrected when spotted. (contributed by Derek Bergin)</span> | # <span style="font-family: Arial; font-size: 16px">Automated tests assume that something in the test environment is immutable, i.e. will not change, because the setup is assumed to be under the testers' control and deviations can be corrected when spotted. (contributed by Derek Bergin)</span> | ||
| − | =<span style="font-family: Arial; font-size: 16px">Questions</span>= | + | =<span style="font-family: Arial; font-size: 16px">'''Questions'''</span>= |
<span style="font-family: Arial; font-size: 16px">Do the tests fail whenever a preceding test was also run? Or whenever a preceding test was not run?</span><br /> <span style="font-family: Arial; font-size: 16px">What runs at the same time as the automated tests?</span><br /> <span style="font-family: Arial; font-size: 16px">Did IT-Support run some updates just as the automated tests were running?</span><br /> <span style="font-family: Arial; font-size: 16px">Was somebody working on the same database as the automated tests just as they were running?</span><br /> <span style="font-family: Arial; font-size: 16px">Have the automated tests been sufficiently tested?</span><br /> <span style="font-family: Arial; font-size: 16px">Who uses which machines and when? Is it possible to plan when each group has access to the machines?</span><br /> <span style="font-family: Arial; font-size: 16px">Is there a central database for everybody or does each user or group have its own?</span><br /> <span style="font-family: Arial; font-size: 16px">What are you depending on not changing in the test environment (what do you assume is immutable)? How will you monitor changes in the test environment (expected and unexpected)? Can you investigate the environment after a failure?</span> | <span style="font-family: Arial; font-size: 16px">Do the tests fail whenever a preceding test was also run? Or whenever a preceding test was not run?</span><br /> <span style="font-family: Arial; font-size: 16px">What runs at the same time as the automated tests?</span><br /> <span style="font-family: Arial; font-size: 16px">Did IT-Support run some updates just as the automated tests were running?</span><br /> <span style="font-family: Arial; font-size: 16px">Was somebody working on the same database as the automated tests just as they were running?</span><br /> <span style="font-family: Arial; font-size: 16px">Have the automated tests been sufficiently tested?</span><br /> <span style="font-family: Arial; font-size: 16px">Who uses which machines and when? Is it possible to plan when each group has access to the machines?</span><br /> <span style="font-family: Arial; font-size: 16px">Is there a central database for everybody or does each user or group have its own?</span><br /> <span style="font-family: Arial; font-size: 16px">What are you depending on not changing in the test environment (what do you assume is immutable)? How will you monitor changes in the test environment (expected and unexpected)? Can you investigate the environment after a failure?</span> | ||
| − | =<span style=" | + | =<span style="font-size: 16px">'''Resolving Patterns'''</span>= |
| − | span style=" | + | <span style="font-size: 16px">The following patterns can all be recommended:</span><br /> |
| − | |||
* <span style="font-size: 16px">[[ DEDICATED RESOURCES]]: this is the pattern of choice for this issue</span> | * <span style="font-size: 16px">[[ DEDICATED RESOURCES]]: this is the pattern of choice for this issue</span> | ||
* <span style="font-size: 16px">[[ FAIL GRACEFULLY]]: use this pattern if you suspect your issue to be like Example 7.</span> | * <span style="font-size: 16px">[[ FAIL GRACEFULLY]]: use this pattern if you suspect your issue to be like Example 7.</span> | ||
Revision as of 18:34, 29 April 2018
.................................................................................................................Main Page / Back to Execution Issues / Back to Test Automation Issues
.................................................................................................................Main Page / Back to Execution Issues / Back to Test Automation Issues
Issue summary
The automated tests seem to pass or fail randomly
Category
Execution
Examples
- Mostly the tests pass but once in a while they fail. If they are rerun they pass as usual
- Automated tests fail because development was using your same database and changed its structure
- Testers changed the files test automation uses to compare results and now those tests don't pass any longer
- IT-Support shut down your machine to install some new software and killed your automation run
- IT-Support installed new software and now your tool doesn't work any more
- Automated tests run in parallel using the same databases and mess up each others data
- A previous test failed and left the system "dirty" so that all successive tests also failed
- Automated tests assume that something in the test environment is immutable, i.e. will not change, because the setup is assumed to be under the testers' control and deviations can be corrected when spotted. (contributed by Derek Bergin)
Questions
Do the tests fail whenever a preceding test was also run? Or whenever a preceding test was not run?
What runs at the same time as the automated tests?
Did IT-Support run some updates just as the automated tests were running?
Was somebody working on the same database as the automated tests just as they were running?
Have the automated tests been sufficiently tested?
Who uses which machines and when? Is it possible to plan when each group has access to the machines?
Is there a central database for everybody or does each user or group have its own?
What are you depending on not changing in the test environment (what do you assume is immutable)? How will you monitor changes in the test environment (expected and unexpected)? Can you investigate the environment after a failure?
Resolving Patterns
The following patterns can all be recommended:
- DEDICATED RESOURCES: this is the pattern of choice for this issue
- FAIL GRACEFULLY: use this pattern if you suspect your issue to be like Example 7.
- INDEPENDENT TEST CASES: use this pattern if your test automation suite is tripping itself up
- RIGHT INTERACTION LEVEL: make sure that your tests interact with the SUT at the most effective level
- TEST THE TESTS: this is really a no-brainer. Use it always
- VERIFY-ACT-VERIFY: use this pattern if you cannot apply FRESH SETUP
- If different execution times are the reason for the 'flakiness' use the pattern VARIABLE DELAYS
.................................................................................................................Main Page / Back to Execution Issues / Back to Test Automation Issues