KNOW WHEN TO STOP

.................................................................................................................Main Page / Back to Management Patterns / Back to Test Automation Patterns

Pattern Summary

Not all test cases can or should be automated

Category

Management

Context

This pattern is most important if you want to implement test automation efficiently, so it is useful even if you just want to write disposable scripts

Description

Not all test cases can or should be automated. The following test types are better handled manually:

• User interface and usability tests
• Tests that run only once in a while
• Tests for a system that is still very unstable
• Tests that take to long to automate (not worth the effort)

When you are automating tests (just as when you are testing), there will be many things that you could automate (test) - a virtually infinite list. But your time will always be limited, so it is getting the balance right between not automating enough and losing benefits you could have, and automating too much and getting decreasing returns from additional automated tests. When to stop is at the point where more tests wouldn't give you enough additional value.

Some tests are very difficult to automate, and the time spent on automating them would be better spent testing this particular thing manually and automating other things. For example, what if you are testing "Captcha" - the wavy letters that you need to type in to a web site to prove that you are a human being? The point of Captcha is that it shouldn't be possible for a computer to do this, so if you could automate the tests for it, it would actually prove that it didn't work! Yes, there are ways to test Captcha automatically, but if this needs to be tested only a few times, it would be better to test it manually, and automate things that are more straight-forward.

Implementation

If you automated the most important tests first, in order of value, then whenever you stop, you will have automated the best tests that you could. So prioritize what tests are most important to automate.

Potential problems

If you stop automating too soon, you will miss out on potential benefits of automation; if you keep automating after you should have stopped, you will be wasting time and effort without good benefits. As with many situations in automation, there is not a clear-cut boundary for when is best to stop.

Issues addressed by this pattern

STALLED AUTOMATION
UNAUTOMATABLE TEST CASES
UNFOCUSED AUTOMATION

Experiences

If you have used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.

.................................................................................................................Main Page / Back to Management Patterns / Back to Test Automation Patterns