Difference between revisions of "KILL THE ZOMBIES"

From Test Automation Patterns
Jump to navigation Jump to search
 
(2 intermediate revisions by 2 users not shown)
Line 10: Line 10:
 
=<span style="font-size: 16px">'''Implementation'''</span>=
 
=<span style="font-size: 16px">'''Implementation'''</span>=
 
<span style="font-size: 16px">Check regularly for test cases, scripts or data that aren’t used any longer and remove them from the system.</span><br /> <span style="font-size: 16px">Be suspicious if tests seem to be passing too easily. If you are "taking over" existing automated tests from someone else, make sure you know what all of the tests are actually doing.</span><br /> <span style="font-size: 16px">Write some scripts or use monitoring tools to find out what is going on.</span>  
 
<span style="font-size: 16px">Check regularly for test cases, scripts or data that aren’t used any longer and remove them from the system.</span><br /> <span style="font-size: 16px">Be suspicious if tests seem to be passing too easily. If you are "taking over" existing automated tests from someone else, make sure you know what all of the tests are actually doing.</span><br /> <span style="font-size: 16px">Write some scripts or use monitoring tools to find out what is going on.</span>  
=<span style="font-size: 16px">'''Possible problems'''</span>=
+
=<span style="font-size: 16px">'''Potential problems'''</span>=
 
<span style="font-size: 16px">If your tool doesn’t support you here, use the "find in files" function of a simple text editor to check what is actually still in use.</span>  
 
<span style="font-size: 16px">If your tool doesn’t support you here, use the "find in files" function of a simple text editor to check what is actually still in use.</span>  
 
=<span style="font-size: 16px">'''Issues addressed by this pattern'''</span>=
 
=<span style="font-size: 16px">'''Issues addressed by this pattern'''</span>=
 
<span style="font-size: 16px">''[[DATA CREEP]]''</span><br /> ''<span style="font-size: 16px">[[ OBSCURE TESTS]]</span>''<br /> ''<span style="font-size: 16px">[[ SCRIPT CREEP]]</span>''  
 
<span style="font-size: 16px">''[[DATA CREEP]]''</span><br /> ''<span style="font-size: 16px">[[ OBSCURE TESTS]]</span>''<br /> ''<span style="font-size: 16px">[[ SCRIPT CREEP]]</span>''  
 
=<span style="font-size: 16px">'''Experiences'''</span>=
 
=<span style="font-size: 16px">'''Experiences'''</span>=
<span style="font-size: 16px">The lovely term "zombie tests" comes from Chapter 15, Test Automation of SAP Business Processes, by Christophe Mecke, Malanie Reinwarth and Armin Gienger. They discuss why they use static analysis tools on their automation script code:</span><br /> <br /> <span style="font-size: 16px">"Our favourite checks are on test script modules that are not being used anywhere and test scripts with no code at all. You might have doubts if this is really relevant and why we even bother with these checks, but we found that this is really an important and unfortunately existing issue in practice: What sometimes happens is that a test script runs into a problem and it turns out that the problem is not on the product side but in the test script itself. Unfortunately, sometimes the erroneous code in the test script is commented out for whatever reason (e.g. to correct it at a later time). If this code is forgotten, you have created a "zombie" test case. From now on, it appears in all test reports with a nice green status (but actually doing nothing). Because nobody usually looks into a test script that is running fine, the test case becomes completely invisible. This can lead to a dangerous situation and might even result in wrong release decisions. A static check on this brings this script back on our radar screen so that it can be fixed."</span><br />  - <span style="font-size: 16px">From the book Experiences of Test Automation (Dorothy Graham and Mark Fewster, Pearson, 2012), pages 285-6.</span><br /> <br /> <span style="font-size: 16px">Tim Fretwell tells of a completely broken build (left out the main part) but three automated tests still passed.</span><br /> <br /> <br /> <span style="font-size: 16px">If you have used this pattern, please add your name and a brief story of how you used this pattern: your context, what you did, and how well it worked - or how it didn't work!</span><br /> <br /> <span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Process Patterns]] / Back to [[Test Automation Patterns]]</span>
+
<span style="font-size: 16px">The lovely term "zombie tests" comes from Chapter 15, Test Automation of SAP Business Processes, by Christophe Mecke, Malanie Reinwarth and Armin Gienger. They discuss why they use static analysis tools on their automation script code:</span><br /> <br /> <span style="font-size: 16px">"Our favourite checks are on test script modules that are not being used anywhere and test scripts with no code at all. You might have doubts if this is really relevant and why we even bother with these checks, but we found that this is really an important and unfortunately existing issue in practice: What sometimes happens is that a test script runs into a problem and it turns out that the problem is not on the product side but in the test script itself. Unfortunately, sometimes the erroneous code in the test script is commented out for whatever reason (e.g. to correct it at a later time). If this code is forgotten, you have created a "zombie" test case. From now on, it appears in all test reports with a nice green status (but actually doing nothing). Because nobody usually looks into a test script that is running fine, the test case becomes completely invisible. This can lead to a dangerous situation and might even result in wrong release decisions. A static check on this brings this script back on our radar screen so that it can be fixed."</span><br />  - <span style="font-size: 16px">From the book Experiences of Test Automation (Dorothy Graham and Mark Fewster, Pearson, 2012), pages 285-6.</span><br /> <br /> <span style="font-size: 16px">Tim Fretwell tells of a completely broken build (left out the main part) but three automated tests still passed.</span><br /> <br /> <br />  
 +
 
 +
<span style="font-size: 16px">If you have also used this pattern and would like to contribute your experience to the wiki, please go to [[Feedback]] to submit your experience or comment.</span><br /> <br />  
 +
 
 +
 
 +
<span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Process Patterns]] / Back to [[Test Automation Patterns]]</span>

Latest revision as of 14:38, 21 August 2018

.................................................................................................................Main Page / Back to Process Patterns / Back to Test Automation Patterns

Pattern summary

Regularly delete or archive test cases, scripts or test data that are not used. Make sure that tests are actually testing something.

Category

Process

Context

This pattern should be applied to maintain long lasting automation

Description

Regularly archive test cases, scripts or test data that are not used. Make sure that tests are actually testing something.

Implementation

Check regularly for test cases, scripts or data that aren’t used any longer and remove them from the system.
Be suspicious if tests seem to be passing too easily. If you are "taking over" existing automated tests from someone else, make sure you know what all of the tests are actually doing.
Write some scripts or use monitoring tools to find out what is going on.

Potential problems

If your tool doesn’t support you here, use the "find in files" function of a simple text editor to check what is actually still in use.

Issues addressed by this pattern

DATA CREEP
OBSCURE TESTS
SCRIPT CREEP

Experiences

The lovely term "zombie tests" comes from Chapter 15, Test Automation of SAP Business Processes, by Christophe Mecke, Malanie Reinwarth and Armin Gienger. They discuss why they use static analysis tools on their automation script code:

"Our favourite checks are on test script modules that are not being used anywhere and test scripts with no code at all. You might have doubts if this is really relevant and why we even bother with these checks, but we found that this is really an important and unfortunately existing issue in practice: What sometimes happens is that a test script runs into a problem and it turns out that the problem is not on the product side but in the test script itself. Unfortunately, sometimes the erroneous code in the test script is commented out for whatever reason (e.g. to correct it at a later time). If this code is forgotten, you have created a "zombie" test case. From now on, it appears in all test reports with a nice green status (but actually doing nothing). Because nobody usually looks into a test script that is running fine, the test case becomes completely invisible. This can lead to a dangerous situation and might even result in wrong release decisions. A static check on this brings this script back on our radar screen so that it can be fixed."
- From the book Experiences of Test Automation (Dorothy Graham and Mark Fewster, Pearson, 2012), pages 285-6.

Tim Fretwell tells of a completely broken build (left out the main part) but three automated tests still passed.


If you have also used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.


.................................................................................................................Main Page / Back to Process Patterns / Back to Test Automation Patterns