.................................................................................................................Main Page / Back to Execution Issues / Back to Test Automation Issues

Issue summary

The tests fail not because of errors in the SUT, but because of errors in the test automation testware or environment issues

Examples

Tests fail because a window pops up that was not considered when developing the automation
Tests fail because the initial conditions have been corrupted by another test
A test always passes if it runs by itself but always fails when run in the test suite
Tests fail because something else is running on the same machine
Tests fail because the database has been corrupted or changed by another application
Tests fail because you are using a SENSITIVE COMPARE and so changes that have nothing to do with your test case affect the results

Are the initial conditions set correctly?
Are there dependencies between the test cases?
Have the tests been sufficiently tested?

Most recommended:

COMPARISON DESIGN: design the comparison of test results to be as efficient as possible, balancing Dynamic and Post-Execution Comparison, and using a mixture of Sensitive and Robust/ Specific comparisons.
DEDICATED RESOURCES: Use this pattern if you have issues similar to Examples 4. or 5.
FRESH SETUP: This pattern guards against issues like Example 2.
INDEPENDENT TEST CASES: apply this pattern to get rid of issues like Example 3.
RIGHT INTERACTION LEVEL: make sure that your tests interact with the SUT at the most effective level

Other useful patterns:

MAINTAIN THE TESTWARE: apply this pattern to make sure that your automation keeps working even is the SUT is being changed
SHARE INFORMATION: this pattern helps you learn in time when development is planning big or small changes that affect automation
SPECIFIC COMPARE: Expected results are specific to the test case so changes to objects not processed in the test case don't affect the test results. Use this pattern for issues like Example 6
TEST THE TESTS: use this pattern always (even for quick fixes!)

.................................................................................................................Main Page / Back to Execution Issues Execution Issues / Back to Test Automation Issues