FLAKY TESTS
Issue summary
The automated tests seem to pass or fail randomly
Category
Execution
Examples
- Mostly the tests pass but once in a while they fail. If they are rerun they pass as usual
- Automated tests fail because development was using your same database and changed its structure
- Testers changed the files test automation uses to compare results and now those tests don't pass any longer
- IT-Support shut down your machine to install some new software and killed your automation run
- IT-Support installed new software and now your tool doesn't work any more
- Automated tests run in parallel using the same databases and mess up each other's data
- A previous test failed and left the system "dirty" so that all successive tests also failed
- Automated tests assume that something in the test environment is immutable, i.e. will not change, because the setup is assumed to be under the testers' control and deviations can be corrected when spotted. (contributed by Derek Bergin)
Questions
Do the tests fail whenever a preceding test was also run? Or whenever a preceding test was not run?
What runs at the same time as the automated tests?
Did IT-Support run some updates just as the automated tests were running?
Was somebody working on the same database as the automated tests just as they were running?
Have the automated tests been sufficiently tested?
Who uses which machines and when? Is it possible to plan when each group has access to the machines?
Is there a central database for everybody or does each user or group have its own?
What are you depending on not changing in the test environment (what do you assume is immutable)? How will you monitor changes in the test environment (expected and unexpected)? Can you investigate the environment after a failure?
Resolving Patterns
The following patterns can all be recommended:
- DEDICATED RESOURCES: this is the pattern of choice for this issue
- FAIL GRACEFULLY: use this pattern if you suspect your issue to be like Example 7.
- INDEPENDENT TEST CASES: use this pattern if your test automation suite is tripping itself up
- RIGHT INTERACTION LEVEL: make sure that your tests interact with the SUT at the most effective level
- TEST THE TESTS: this is really a no-brainer. Use it always
- VERIFY-ACT-VERIFY: use this pattern if you cannot apply FRESH SETUP
- If different execution times are the reason for the 'flakiness' use the pattern VARIABLE DELAYS
.................................................................................................................Main Page / Back to Execution Issues / Back to Test Automation Issues