Difference between revisions of "KILL THE ZOMBIES"
| (2 intermediate revisions by 2 users not shown) | |||
| Line 10: | Line 10: | ||
=<span style="font-size: 16px">'''Implementation'''</span>= | =<span style="font-size: 16px">'''Implementation'''</span>= | ||
<span style="font-size: 16px">Check regularly for test cases, scripts or data that aren’t used any longer and remove them from the system.</span><br /> <span style="font-size: 16px">Be suspicious if tests seem to be passing too easily. If you are "taking over" existing automated tests from someone else, make sure you know what all of the tests are actually doing.</span><br /> <span style="font-size: 16px">Write some scripts or use monitoring tools to find out what is going on.</span> | <span style="font-size: 16px">Check regularly for test cases, scripts or data that aren’t used any longer and remove them from the system.</span><br /> <span style="font-size: 16px">Be suspicious if tests seem to be passing too easily. If you are "taking over" existing automated tests from someone else, make sure you know what all of the tests are actually doing.</span><br /> <span style="font-size: 16px">Write some scripts or use monitoring tools to find out what is going on.</span> | ||
| − | =<span style="font-size: 16px">''' | + | =<span style="font-size: 16px">'''Potential problems'''</span>= |
<span style="font-size: 16px">If your tool doesn’t support you here, use the "find in files" function of a simple text editor to check what is actually still in use.</span> | <span style="font-size: 16px">If your tool doesn’t support you here, use the "find in files" function of a simple text editor to check what is actually still in use.</span> | ||
=<span style="font-size: 16px">'''Issues addressed by this pattern'''</span>= | =<span style="font-size: 16px">'''Issues addressed by this pattern'''</span>= | ||
<span style="font-size: 16px">''[[DATA CREEP]]''</span><br /> ''<span style="font-size: 16px">[[ OBSCURE TESTS]]</span>''<br /> ''<span style="font-size: 16px">[[ SCRIPT CREEP]]</span>'' | <span style="font-size: 16px">''[[DATA CREEP]]''</span><br /> ''<span style="font-size: 16px">[[ OBSCURE TESTS]]</span>''<br /> ''<span style="font-size: 16px">[[ SCRIPT CREEP]]</span>'' | ||
=<span style="font-size: 16px">'''Experiences'''</span>= | =<span style="font-size: 16px">'''Experiences'''</span>= | ||
| − | <span style="font-size: 16px">The lovely term "zombie tests" comes from Chapter 15, Test Automation of SAP Business Processes, by Christophe Mecke, Malanie Reinwarth and Armin Gienger. They discuss why they use static analysis tools on their automation script code:</span><br /> <br /> <span style="font-size: 16px">"Our favourite checks are on test script modules that are not being used anywhere and test scripts with no code at all. You might have doubts if this is really relevant and why we even bother with these checks, but we found that this is really an important and unfortunately existing issue in practice: What sometimes happens is that a test script runs into a problem and it turns out that the problem is not on the product side but in the test script itself. Unfortunately, sometimes the erroneous code in the test script is commented out for whatever reason (e.g. to correct it at a later time). If this code is forgotten, you have created a "zombie" test case. From now on, it appears in all test reports with a nice green status (but actually doing nothing). Because nobody usually looks into a test script that is running fine, the test case becomes completely invisible. This can lead to a dangerous situation and might even result in wrong release decisions. A static check on this brings this script back on our radar screen so that it can be fixed."</span><br /> - <span style="font-size: 16px">From the book Experiences of Test Automation (Dorothy Graham and Mark Fewster, Pearson, 2012), pages 285-6.</span><br /> <br /> <span style="font-size: 16px">Tim Fretwell tells of a completely broken build (left out the main part) but three automated tests still passed.</span><br /> <br /> <br /> <span style="font-size: 16px">If you have used this pattern, please | + | <span style="font-size: 16px">The lovely term "zombie tests" comes from Chapter 15, Test Automation of SAP Business Processes, by Christophe Mecke, Malanie Reinwarth and Armin Gienger. They discuss why they use static analysis tools on their automation script code:</span><br /> <br /> <span style="font-size: 16px">"Our favourite checks are on test script modules that are not being used anywhere and test scripts with no code at all. You might have doubts if this is really relevant and why we even bother with these checks, but we found that this is really an important and unfortunately existing issue in practice: What sometimes happens is that a test script runs into a problem and it turns out that the problem is not on the product side but in the test script itself. Unfortunately, sometimes the erroneous code in the test script is commented out for whatever reason (e.g. to correct it at a later time). If this code is forgotten, you have created a "zombie" test case. From now on, it appears in all test reports with a nice green status (but actually doing nothing). Because nobody usually looks into a test script that is running fine, the test case becomes completely invisible. This can lead to a dangerous situation and might even result in wrong release decisions. A static check on this brings this script back on our radar screen so that it can be fixed."</span><br /> - <span style="font-size: 16px">From the book Experiences of Test Automation (Dorothy Graham and Mark Fewster, Pearson, 2012), pages 285-6.</span><br /> <br /> <span style="font-size: 16px">Tim Fretwell tells of a completely broken build (left out the main part) but three automated tests still passed.</span><br /> <br /> <br /> |
| + | |||
| + | <span style="font-size: 16px">If you have also used this pattern and would like to contribute your experience to the wiki, please go to [[Feedback]] to submit your experience or comment.</span><br /> <br /> | ||
| + | |||
| + | |||
| + | <span style="font-size: 14px">.................................................................................................................[[Main Page]] / Back to [[Process Patterns]] / Back to [[Test Automation Patterns]]</span> | ||
Latest revision as of 14:38, 21 August 2018
Pattern summary
Regularly delete or archive test cases, scripts or test data that are not used. Make sure that tests are actually testing something.
Category
Process
Context
This pattern should be applied to maintain long lasting automation
Description
Regularly archive test cases, scripts or test data that are not used. Make sure that tests are actually testing something.
Implementation
Check regularly for test cases, scripts or data that aren’t used any longer and remove them from the system.
Be suspicious if tests seem to be passing too easily. If you are "taking over" existing automated tests from someone else, make sure you know what all of the tests are actually doing.
Write some scripts or use monitoring tools to find out what is going on.
Potential problems
If your tool doesn’t support you here, use the "find in files" function of a simple text editor to check what is actually still in use.
Issues addressed by this pattern
DATA CREEP
OBSCURE TESTS
SCRIPT CREEP
Experiences
The lovely term "zombie tests" comes from Chapter 15, Test Automation of SAP Business Processes, by Christophe Mecke, Malanie Reinwarth and Armin Gienger. They discuss why they use static analysis tools on their automation script code:
"Our favourite checks are on test script modules that are not being used anywhere and test scripts with no code at all. You might have doubts if this is really relevant and why we even bother with these checks, but we found that this is really an important and unfortunately existing issue in practice: What sometimes happens is that a test script runs into a problem and it turns out that the problem is not on the product side but in the test script itself. Unfortunately, sometimes the erroneous code in the test script is commented out for whatever reason (e.g. to correct it at a later time). If this code is forgotten, you have created a "zombie" test case. From now on, it appears in all test reports with a nice green status (but actually doing nothing). Because nobody usually looks into a test script that is running fine, the test case becomes completely invisible. This can lead to a dangerous situation and might even result in wrong release decisions. A static check on this brings this script back on our radar screen so that it can be fixed."
- From the book Experiences of Test Automation (Dorothy Graham and Mark Fewster, Pearson, 2012), pages 285-6.
Tim Fretwell tells of a completely broken build (left out the main part) but three automated tests still passed.
If you have also used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.
.................................................................................................................Main Page / Back to Process Patterns / Back to Test Automation Patterns