TEST SELECTOR
Pattern summary
Implement your test cases so that you can turn on various selection criteria for whether or not you include a given test in an execution run.
Category
Design
Context
This pattern is needed when you have a lot of automated tests, when you can no longer run all of them in the allotted time.
Description
The need for this pattern is not obvious when you first start automating, but it is an important one to take into consideration if you want to have large-scale sustainable automation.
Implementation
As part of the description or documentation of a test, include some Tags to identify this particular test in different ways. For example, as part of a Smoke Test, a regression test for a particular feature, a depth test for a function, when the test last found a bug, even the test's author. Choose your tags depending on the different ways you might like to form subsets of tests for execution.
When you DOCUMENT THE TESTWARE, this is one of the things that you will SET STANDARDS for.
When you are gathering tests for an execution run, your TEST AUTOMATION FRAMEWORK should enable you to choose the tests to run by specifying the selector tags to include. For example for tonight's overnight run, you may want to execute:
- the tests that failed last time they were run (to see if they are fixed correctly)
- the depth tests for the function that was most extensively changed
- the highest priority tests from the full standard smoke test
- Leslie's tests, as Leslie has been off sick for a few days
Note that your framework needs to be designed to enable the tests to be selected by these Test Selectors!
The sets of tests that you choose will be determined by how you PRIORITIZE TESTS.
Potential problems
If this is not considered at the beginning of an automation effort, it is more difficult to implement, but is usually still worth doing, for the flexibility it gives to test execution.
The format of the Test Selector needs to be clearly defined and the standards for their definition and use enforced.
Issues addressed by this pattern
INEFFICIENT EXECUTION
INFLEXIBLE AUTOMATION
Experiences
From Marcus Merrell:
Here's the list of test selectors that we use. Some of it is specific to our domain, but it should give a pretty good idea of how the test suite can be divided horizontally, vertically, or by priority, scope, or any other way you'd want to weight it:
SMOKE
REGRESSION
PROD_SAFE
POP_UP
SEARCH
ADMIN
FACEBOOK
LONG_RUNNING
LOGIN
NEW_CMS
SIGN_UP
COMMUNITY
MOBILE_WEB
TABLET
MY_ACCOUNT
DESKTOP_ANALYTICS
TABLET_ANALYTICS
MOBILE_ANALYTICS
PRIVATE_API
DATA_DRIVEN
PROD_MONITOR
INQUISITION
If you have also used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.