AUTOMATE GOOD TESTS

.................................................................................................................Main Page / Back to Design Patterns / Back to Test Automation Patterns

Pattern summary

Automate only the tests that bring the most Return on Investment (ROI).

Category

Design

Context

This pattern is useful when you have to decide what to automate, so you should apply it not only when starting with test automation from scratch, but also every time you want to increase your automation

Description

Automate only the tests that bring the most return on investment. Of course first of all you must know your test cases (remember the warning by Dorothy Graham: Automating chaos just gives faster chaos). You may need to rework them first.

Implementation

Good candidates for automation are:

• Smoke tests: since they are run very often, they are also profitable fast
• Regression tests: as manual tests they are often boring and time-consuming and so are either performed poorly or not at all. They are well suited for automation since they usually test stable parts of the SUT and so the expected maintenance effort is reasonably small.
• Identical tests for different environments: automating such tests also pays off fast since the most effort is usually spent in automating the first environment.
• Complex tests: if manual testing is too difficult, it will not be performed.
• Tests that require machine precision
• Repetitive, boring and time-consuming processing: set-ups for manual testing can also be automated

Recommendations

• Remember the 80/20 rule: 20 % of the application accounts for ca 80% of the usage. Automate the 20% only.
• Tests that cover critical areas in the Software Under Test (SUT) should be automated first.
• Not every manual test is suitable for automation, start with the most repetitive.
• Not every regression test should be automated from the start. It makes sense to start with the less complicated and proceed later to the more difficult ones.
• Don't automate only “strategic” applications, automate also the stuff that's boring to execute and could easily be automated [1]. Vincent Wijnen noticed that by automating the most boring test cases the number of testers leaving the company fell noticeably!
• If more than one team is doing automation make sure that they don't automate the same tests.

Issues addressed by this pattern

HIGH ROI EXPECTATIONS
TOO EARLY AUTOMATION

Experiences

Bryan Bakker* says:I have participated in several projects where the reliability of the basic functions of a product were important attention points. These basic functions worked most of the time, but not always... When test automation started on these projects, I started with automating reliability tests. Some very simple and basic tests, which are executed over and over again to discover reliability issues:

• Very easy to automate these test cases
• Almost impossible to execute manually
• Numerous reliability issues which have already been reported by clients, could be reproduced by this approach, making it possible for developers to analyze and fix these problems

In these projects the right tests to automate were reliability tests.
As a secondary result the automation approach proved to be adding value and time and money became available to automate more tests (regression, smoke, performance).

Note: Bryan Bakker, along with colleagues Rob de Bie, Rene van den Eertwegh and Peter Wijhhoven, published a book in 2015 called: Finally … Reliable Software! A practical approach to design for reliability". This book explains reliability theory in an accessible way and uses an example case study to show how to do it.

If you have also used this pattern, please add your name and a brief story of how you used this pattern: your context, what you did, and how well it worked - or how it didn't work!

.................................................................................................................Main Page / Back to Design Patterns / Back to Test Automation Patterns

[1]] Suggested at SIGIST Workshop 5 Dec 2013