Difference between revisions of "HARD-TO-AUTOMATE"
Jump to navigation
Jump to search
m (empty lines removed) |
|||
| Line 18: | Line 18: | ||
# <span style="font-size: 16px">Unexpected pop-ups</span> | # <span style="font-size: 16px">Unexpected pop-ups</span> | ||
# <span style="font-size: 16px">The SUT is very slow so it takes a lot of time to test the automation scripts</span> | # <span style="font-size: 16px">The SUT is very slow so it takes a lot of time to test the automation scripts</span> | ||
| + | # <span style="font-size: 16px">APIs or AI deliver non-predictable results</span> | ||
=<span style="font-size: 16px">'''Questions'''</span>= | =<span style="font-size: 16px">'''Questions'''</span>= | ||
<span style="font-family: Arial; font-size: 16px">Where does the complexity lie? In preparing the initial conditions? Test execution? Checking the results? Dependencies?</span><br /> <span style="font-family: Arial; font-size: 16px">Do you have the right resources? What do you need?</span><br /> <span style="font-family: Arial; font-size: 16px">Can you simulate hardware? or software?</span><br /> <span style="font-size: 16px">Has the current tool been selected especially for the SUT or has it been "inherited" from previous applications?</span><br /> <span style="font-size: 16px">Do the developers know about the automation problems? Do they care?</span><br /> <span style="font-size: 16px">Do the developers have guidelines/standards for building in 'testability'? If not, why not?.</span><br /> <span style="font-size: 16px">How do manual testers check the results?</span><br /> <span style="font-size: 16px">How complex are the test cases?</span><br /> <span style="font-size: 16px">How do you collect the necessary data?</span> | <span style="font-family: Arial; font-size: 16px">Where does the complexity lie? In preparing the initial conditions? Test execution? Checking the results? Dependencies?</span><br /> <span style="font-family: Arial; font-size: 16px">Do you have the right resources? What do you need?</span><br /> <span style="font-family: Arial; font-size: 16px">Can you simulate hardware? or software?</span><br /> <span style="font-size: 16px">Has the current tool been selected especially for the SUT or has it been "inherited" from previous applications?</span><br /> <span style="font-size: 16px">Do the developers know about the automation problems? Do they care?</span><br /> <span style="font-size: 16px">Do the developers have guidelines/standards for building in 'testability'? If not, why not?.</span><br /> <span style="font-size: 16px">How do manual testers check the results?</span><br /> <span style="font-size: 16px">How complex are the test cases?</span><br /> <span style="font-size: 16px">How do you collect the necessary data?</span> | ||
| Line 26: | Line 27: | ||
* <span style="font-size: 16px">[[ TAKE SMALL STEPS]]: use this pattern to break up the problems in more chewable chunks</span> | * <span style="font-size: 16px">[[ TAKE SMALL STEPS]]: use this pattern to break up the problems in more chewable chunks</span> | ||
* <span style="font-size: 16px">[[ TESTABLE SOFTWARE]]</span>: <span style="font-size: 16px">use this pattern for issues similar to Examples 1. 2. and 3.</span> | * <span style="font-size: 16px">[[ TESTABLE SOFTWARE]]</span>: <span style="font-size: 16px">use this pattern for issues similar to Examples 1. 2. and 3.</span> | ||
| − | * <span style="font-size: 16px">[[ THINK OUT-OF-THE-BOX]]: try to look at the problem from unusual viewpoints</span> | + | * <span style="font-size: 16px">[[ THINK OUT-OF-THE-BOX]]: try to look at the problem from unusual viewpoints. This is especially important for issues like Example 13</span> |
* <span style="font-size: 16px">[[VARIABLE DELAYS]]: use this pattern for issues Like Example 4</span> | * <span style="font-size: 16px">[[VARIABLE DELAYS]]: use this pattern for issues Like Example 4</span> | ||
<br /> <span style="font-size: 16px">Other useful patterns:</span><br /> | <br /> <span style="font-size: 16px">Other useful patterns:</span><br /> | ||
Latest revision as of 15:57, 27 June 2018
.................................................................................................................Main Page / Back to Design Issues / Back to Test Automation Issues
Points 1 and 2 in Examples are contributed by Jim Hazen, Point 10 by Thorsten Schönfelder.
Issue summary
The Software Under Test (SUT) doesn’t support automated testing very well or not at all.
Category
Design
Examples
The problem range is:
- Non-identifiable (uniquely) GUI-Objects; caused by lack of unique name or other property to make it unique.
- Third Party or Customized GUI-Objects that are not recognized properly or are not 'open' (exposed methods and properties).
- Random processing
- Fluctuating response times.
- The expected results change after each new version of the SUT
- Different systems (hardware or software) that interact in some way.
- Tests have to run on a steadily growing number of browsers or environments
- Test results consist in entries in many database tables or files related to each other.
- For embedded systems or mobile devices results are difficult to check because of timing issues or not very visible results (e.g. intermediate results)
- Test data is hard to create due to the number of systems involved or a limited refresh cycle.
- Unexpected pop-ups
- The SUT is very slow so it takes a lot of time to test the automation scripts
- APIs or AI deliver non-predictable results
Questions
Where does the complexity lie? In preparing the initial conditions? Test execution? Checking the results? Dependencies?
Do you have the right resources? What do you need?
Can you simulate hardware? or software?
Has the current tool been selected especially for the SUT or has it been "inherited" from previous applications?
Do the developers know about the automation problems? Do they care?
Do the developers have guidelines/standards for building in 'testability'? If not, why not?.
How do manual testers check the results?
How complex are the test cases?
How do you collect the necessary data?
Resolving Patterns
Most recommended:
- DO A PILOT: use this pattern to find out what the problems are and ways to tackle them
- RIGHT TOOLS: use this pattern if you are confronting issues like Examples 1 or 2.
- TAKE SMALL STEPS: use this pattern to break up the problems in more chewable chunks
- TESTABLE SOFTWARE: use this pattern for issues similar to Examples 1. 2. and 3.
- THINK OUT-OF-THE-BOX: try to look at the problem from unusual viewpoints. This is especially important for issues like Example 13
- VARIABLE DELAYS: use this pattern for issues Like Example 4
Other useful patterns:
- EASY TO DEBUG FAILURES. use this pattern for issues like Example 8.
- GET ON THE CLOUD: this is the pattern of choice for issues like Example 7.
- KEEP IT SIMPLE: this pattern is always helpful
- SHARE INFORMATION: use this pattern for better communication between development, testing and automation (Examples 1, 2, 4., 5., 11. and 12.)
.................................................................................................................Main Page / Back to Design Issues / Back to Test Automation Issues
Points 1 and 2 in Examples are contributed by Jim Hazen, Point 10 by Thorsten Schönfelder.