TESTWARE ARCHITECTURE
Pattern summary
Design the structure of your testware so that your automators and testers can work as efficiently as possible.
Category
Design
Context
This pattern is particularly important for large-scale or long lasting automation.
If you just plan to write a few disposable scripts you will not need it.
Description
There are many artefacts in test automation and all of them need to have a place to "live". Scripts, data sets, expected results, actual results, utilities etc. For efficient working, it is important to ensure that testers and automators know where to find artefacts and can access them quickly and easily as they work. For example, a standard structure for testware permits the information of where artefacts are to be found or placed to be made known to the tools, for example, when tests are to be run or test results are reported. This makes the job of interacting with the tools simpler and less error-prone, because the "where to find it" information is already built into your tools or framework.
Whichever tool you use will have some kind of architecture as the default for that tool. One option is to simply adopt the tool's architecture. Although this may be the easiest option in the beginning, it could have long-term negative consequences, as you will then be "tied" to that tool's current architecture.
It is much better to design the architecture of your tests so that it supports the way you want to use your automation. It is relatively straight-forward to adapt the testware to the needs of the tool (e.g. copy a script to the place where the tool expects to find it) - and this can and should also be automated.
Implementation
When you design your architecture, you need to decide where you will store various testware artefacts, i.e. what filing structure will you use, what naming conventions will you use for files and folders, etc.
It is important to distinguish between the test materials and the test results. The test materials are those artefacts that should be in place before a test is executed, such as the test inputs, expected results, any set-up that needs to be done, required data before a test runs, environment settings, and the description / documentation for that test. Many of these artefacts will be used by more than one test - in this case, they should be accessible to all tests that use them, but there should only be one master copy (except for legitimate different versions). These artefacts must be under configuration control to prevent people over-writing edits, for example.
The test results include everything that is produced by the system or software when a test is executed, including the actual results, log files, difference files (between actual and expected results), etc. There are usually many sets of these for a given test, at least for a test that is run many times. These need to be stored in a different way to the test materials, since one set of materials (for a single test) will produce a new copy of the test results each time it is run. If tests are run daily or more often, these sets of results will soon build up, and you don't want them "clogging" the storage of your tests.
It is worth asking whether you need to keep all of the test results each time - you may want to keep the test log, to prove that the test was run, but if the test passes, then the actual results has exactly matched the expected results, so why keep two copies of the same thing?
Your testware architecture should implement ABSTRACTION LEVELS. Make sure that you design the structure of your scripts so that tool-specific scripts are kept to a minimum. To make your automation accessible to a wide variety of testers, including those who are not programmers, make sure that you enable them to write and run automated tests easily. DOMAIN-DRIVEN TESTING and KEYWORD-DRIVEN TESTING are good ways to implement this.
You will have to have MANAGEMENT SUPPORT to get the time you need to design a long-lasting effective architecture for your automated tests.
A TEST AUTOMATION FRAMEWORK is an implementation of at least part of a testware architecture.
Potential problems
It is not necessarily easy to design a good testware architecture, and it does take effort and time. But it is very worthwhile when done well.
Issues addressed by this pattern
BRITTLE SCRIPTS
INFLEXIBLE AUTOMATION
CAN'T FIND WHAT I WANT
TOO EARLY AUTOMATION
TOOL-DRIVEN AUTOMATION
Experiences
If you have used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.