RIGHT TOOLS

.................................................................................................................Main Page / Back to Management Patterns / Back to Test Automation Patterns

Pattern summary

Select the right tools for your application, environment, people, budget and level of testing.

Category

Management

Context

This pattern is appropriate when your automated tests will be around for a long time, and/or when you are beginning test automation and want to "get off on the right foot".

This pattern is not appropriate for one-off or disposable scripts, or when you are experimenting with different tools just to see what they do.

Be aware that having "the right tool" is not all that important, as long as a tool is workable in your context. The other factors involved in test automation are usually much more important and influential than the choice of tool.

Description

In order to select the right tools you must first of all SET CLEAR GOALS as to what you intend to achieve with them and what not. To do this, you must first SHARE INFORMATION with testers and developers to find out what they actually need from automation. Testers and especially managers can have false expectations: things that would be easy to automate are considered so difficult that they are not even mentioned and things that are really difficult to automate are believed to be just a couple of clicks away!
If you are just starting with test automation then your goals will be just to find some tools that can work with your Software Under Test (SUT).
If you are changing to a different tool, an additional important criterion will be how easily you can migrate your tests from your old tool to the new one.

Implementation

Depending on your budget you can select open source, freeware or commercial tools. There are some crucial issues to consider:

• The tool must be able to do what you acquired it for. For instance a tool to run automation from the GUI must recognize all the objects on your screens to drive your application.The following =Score Card=* can help you check that the tool features all the functionality that you need
  • Platform Support – Support for multiple operating systems, tablets & mobile
  • Technology Support - "multi-compiler" vs. "compiler-specific" test tools;
  • Browser Support - Internet Explorer, Firefox, Google Chrome or any other browser based on web browser controls;
  • Data Source Support - obtain data from text and XML files, Excel worksheets and databases like SQL Server, Oracle and MySQL;
  • Multi-Language Support - localized solutions supporting Unicode;
  • Test Type Support - functional, non-functional and unit (i.e. nUnit & MSTest);
  • Test Approach Support – i.e. Hybrid-Keyword/Data-Driven testing;
  • Results & Reporting Integration – including images, files, databases, XML documents;
  • Test Asset / Object Management - map an object not only by its caption or identifier;
  • Class Identification – GAP analysis of object classes (generic / custom) and associated methods capabilities based on complexity, usage, risk, feasibility and re-usability;
  • Test Scenario Maintenance – manual effort (XPATH/regular expressions), self-maintaining (descriptive programming/fuzzy logic) or script less (DSLs);
  • Continuous Build & Integration / Delivery Integration – with build & delivery solution;
  • Future proofing – external encapsulation of test assets & associated meta data (XAML/xPDL), expandability (API/DLL/. NET), HTTP/WCF/COM/WSD and OCR/IR;
  • License, Support & Maintenance Costs – pricing policy along with any hidden costs.
• Most of the commercial tools on the market have demo versions which you can try on your application. Commercial tool vendors offer workshops to prove that their tool is “just right” for you.
• If you get an open source tool, the tool will most likely be maintained and supported by the open source community, although some tools do have commercial companies offering support.

Some questions to ask:

• How easy is it to learn to use the tool? Is it possible to get training?
• How widespread is the script language of the tool? It will be easier to get skilled people for a more popular script language
• How easy is the migration of your testware from an old tool to the new one?
• How well is the tool supported? How long does the vendor plan to support it? Does the vendor plan to introduce a new tool? If yes, will it be compatible with the old one?
• How high is the probability that with new releases the tool will not support your application as well or not at all (tool regression)? **
• How easy is it to compile a usage list?

To find which tools are available on the market you can:

• Search the internet
• Ask in test automation forums what tools other testers use. Then ask what other testers have experienced with the tools that you are considering
• Attend a conference or seminar that includes a expo with tools. You can get a quick demo at the stand and get an initial idea about a tool.
• Engage a consultant

Once you have some good candidates try them SIDE-BY-SIDE.
Finally don’t forget to PREFER FAMILIAR SOLUTIONS if there are already workable tools in the company, you will save costs and effort because you will profit from the already available experience

Potential problems

• Before you start make sure you have MANAGEMENT SUPPORT. You will need it to get the time and resources necessary to select and introduce a new tool.
• Remember that the objective of test automation is to support testers. Any kind of tool that helps testers do their job better is a test automation tool!
• Set priorities regarding what should be automated first. Since the team members need time to learn to use a tool, you don’t need all the tools at the same time, you could end up paying for licenses that you’ll not be using (yet)!
• How much effort you will need to change tools depends primarily on the test automation architecture that has been or will be implemented. If your current architecture doesn’t support well the changing of the tool, then this is the right time to change your automation structure and implement a TEST AUTOMATION FRAMEWORK.

Issues addressed by this pattern

HARD-TO-AUTOMATE
INADEQUATE TOOLS
MANUAL INTERVENTIONS
NO PREVIOUS TEST AUTOMATION
TOOL DEPENDENCY

Experiences

If you have used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.

.................................................................................................................Main Page / Back to Management Patterns / Back to Test Automation Patterns

---

*) The score card has been suggested by Jonathon Wright. It is taken from 'The Big Picture of Test Automation: Test Trustworthiness' – Alan Page, Microsoft (2012).
**) This question was suggested by Michael Lüttel in a talk at iqnite 2014