AUTOMATE GOOD TESTS
Pattern summary
Automate only the tests that bring the most Return on Investment (ROI).
Category
Design
Context
This pattern is useful when you have to decide what to automate, so you should apply it not only when starting with test automation from scratch, but also every time you want to increase your automation
Description
Automate only the tests that bring the most return on investment. Of course first of all you must know your test cases (remember the warning by Dorothy Graham: Automating chaos just gives faster chaos). You may need to rework them first.
Implementation
Good candidates for automation are:
- Smoke tests: since they are run very often, they are also profitable fast
- Regression tests: as manual tests they are often boring and time-consuming and so are either performed poorly or not at all. They are well suited for automation since they usually test stable parts of the SUT and so the expected maintenance effort is reasonably small.
- Identical tests for different environments: automating such tests also pays off fast since the most effort is usually spent in automating the first environment.
- Complex tests: if manual testing is too difficult, it will not be performed.
- Tests that require machine precision
- Repetitive, boring and time-consuming processing: set-ups for manual testing can also be automated
Recommendations
- Remember the 80/20 rule: 20 % of the application accounts for ca 80% of the usage. Automate the 20% only.
- Tests that cover critical areas in the Software Under Test (SUT) should be automated first.
- Not every manual test is suitable for automation, start with the most repetitive.
- Not every regression test should be automated from the start. It makes sense to start with the less complicated and proceed later to the more difficult ones.
- Don't automate only “strategic” applications, automate also the stuff that's boring to execute and could easily be automated [1]].
- If more than one team is doing automation make sure that they don't automate the same tests.
Issues addressed by this pattern
HIGH ROI EXPECTATIONS
TOO EARLY AUTOMATION
Experiences
Bryan Bakker* says:I have participated in several projects where the reliability of the basic functions of a product were important attention points. These basic functions worked most of the time, but not always... When test automation started on these projects, I started with automating reliability tests. Some very simple and basic tests, which are executed over and over again to discover reliability issues:
- Very easy to automate these test cases
- Almost impossible to execute manually
- Numerous reliability issues which have already been reported by clients, could be reproduced by this approach, making it possible for developers to analyze and fix these problems
As a secondary result the automation approach proved to be adding value and time and money became available to automate more tests (regression, smoke, performance).
Note: Bryan Bakker, along with colleagues Rob de Bie, Rene van den Eertwegh and Peter Wijhhoven, published a book in 2015 called: Finally … Reliable Software! A practical approach to design for reliability". This book explains reliability theory in an accessible way and uses an example case study to show how to do it.
If you have also used this pattern, please add your name and a brief story of how you used this pattern: your context, what you did, and how well it worked - or how it didn't work!
.................................................................................................................Main Page / Back to Design Patterns / Back to Test Automation Patterns
[1]] Suggested at SIGIST Workshop 5 Dec 2013