MAINTAIN THE TESTWARE
Pattern summary
Maintain test automation scripts and test data regularly and from the very beginning.
Category
Process
Context
This pattern must be applied if you want your test automation effort to be long lasting. You may not need it for disposable or one-off scripts.
Description
Test automation scripts and test data need to be kept in step with the systems and software that are being tested; as the production software changes, it may cause scripts to fail or use the wrong data. If the testware is not regularly maintained from the very beginning, the automation will fall into disuse and eventually die.
The biggest factor contributing to excessive maintenance is the structure of the testware, but constant maintenance is still needed throughout the useful life of the automation.
Implementation
Here is some advice for keeping on top of testware maintenance:
- If you find an error in the test automation scripts, correct it as soon as possible.
- If tests are too complicated, simplify them. (KEEP IT SIMPLE).
- If a test case doesn’t pass any longer due to changes in the Software Under Test (SUT), check if it is still valuable and if it is, adjust it accordingly (scripts or data). If it isn’t, KILL THE ZOMBIES.
- Verify occasionally that the returned results really are correct. After some time FALSE PASS has a way of slouching in.
- REFACTOR THE TESTWARE, possibly at a time when code is not changing (you can't refactor tests and change code at the same time!)
- Run all your tests regularly. You will quickly notice what is obsolete and what is not.
- PLAN SUPPORT ACTIVITIES: Plan for regular maintenance as part of the ongoing test automation effort.
- Use DEFAULT DATA when your tests need a lot of data, but only a tiny part of it is really pertinent to the test case
- DOCUMENT THE TESTWARE: good documentation is essential for reuse, reuse is essential for maintainability
Potential problems
You may find that certain maintenance tasks are taking an increasing amount of time. In this case it may be time to set aside time for a major re-structuring and REFACTOR THE TESTWARE. Refactoring is like a spurt of dedicated and concentrated maintenance, but carried out so that future maintenance becomes easier after the restructuring.
It may be hard to justify the time taken to maintain the testware when deadlines are tight and resources are in short supply. But without maintenance, your automation will die, just as a plant dies without water.
Issues addressed by this pattern
DATA CREEP
FALSE FAIL
FALSE PASS
OBSCURE TESTS
SCRIPT CREEP
STALLED AUTOMATION
Experiences
Jochim Van Dorpe writes:
We have a 'no regression'-policy in our sprints. When we end a sprint, no 'old' tests should fail. Cards (i.e. stories) that tend to break old tests need to be fixed by the person who broke them before moving on to the next card.
When a test becomes obsolete due to changing functionalities, we give it the lowest execution priority possible, so from then on our execution system won't execute it anymore . We don't delete the test entirely because we want to keep track of the fact that it existed in the past, and that it was useful in the past. In our test management tool we also mark the test as being obsolete, so the Project Lead (PL) or others who read it don't think that it isn't executed anymore by accident.
If you have also used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.
.................................................................................................................Main Page / Back to Process Patterns / Back to Test Automation Patterns