FRAMEWORK COMPETITION

Failure Pattern summary

Different teams build different automation frameworks and are not able or willing to consolidate them

This failure pattern has been added by Michael Stahl

Context

Large organizations with multiple test automation systems

Description

Large organization frequently end up with more than a single test automation system. The competition is usually not a healthy one and leads to wasted effort and on-going conflicts.
The problems that eventually turn up require different countermeasures.

1. TOOL MUSHROOMING, another failure pattern, is a natural occurrence. It is therefore quite normal that the same pattern will develop in parallel in two (or even more) teams in the same organization; especially when the organization is large and spans across different geographical locations.
   At the beginning (stage 1, 2), there isn’t really a problem. The automation solves localized needs. Many times, the teams developing these automation solutions are not aware that another team is also doing automation work; even if they knew, it would not look like something that calls for attention: the other team is solving their localized needs.
   But as both teams move towards Stage 3, the fact that there are two solutions being built becomes more apparent – if only for the fact that you now have two Test Automation teams on the org chart. When the teams move to Stage 4, there is no way that this duplicity will go unrecognized: program managers need to collect tests results data from two systems, which makes their life difficult.
   So what happens next? It is obvious: Both teams get together and peacefully merge their solutions into one system, implementing the best of both initial solutions…
   You think?
   All I can say is that I did not see this happen.
   What does happen is a long series of fruitless discussions at the engineering level, where each team tries to convince the other – and management – that their code; choice of programming language, design, user Interface, feature set, database etc. is superior to the other solution. Therefore, management should cancel the other solution, and pick theirs.
   While these discussions are going on (for months or years), both teams continue to develop their solution, proudly and surely moving towards Stage 5 – Overload – where both systems will be as much trouble as a solution.
   Solving Patterns:
   • GOOD DEVELOPMENT PROCESS
   • LEARN FROM MISTAKES
   • MAINTAIN THE TESTWARE
   • MANAGEMENT SUPPORT
   • SET STANDARDS
     
2. Another situation is where teams are more coordinated and are aware that a test automation solution is under way. They look at the budding solution, like it, and start feeding requirements for enhancements, pushing the solution fast from Stage 1 to the following stages. Since not all the requests can be accommodated by the limited test automation resources, some type of prioritization takes place. Some requests are implemented, some delayed a bit, and some requests somehow never make it to the releases.
   At a certain point, a team whose requests are constantly being pushed out, will get annoyed and frustrated enough to start their own simple, localized solution… and we are back on the TOOL MUSHROOMING wagon and competition of the 1st variety.
   Solving Patterns:
   • ABSTRACTION LEVELS
   • DOCUMENT THE TESTWARE
   • GOOD DEVELOPMENT PROCESS
   • MAINTAIN THE TESTWARE
   • SET STANDARDS
   • TEST THE TESTS
3. Another reason that will push teams to start their own Stage 1 solutions is when the leading test automation framework is in TOOL MUSHROOMING, Stage 5 for too long. After a year or two of constantly fighting the system’s inefficiency, some teams will get fed up enough to start their own Stage 1 solutions, which will in time, become a competing, Stage 5, inefficient behemoth.
   Solving Patterns:
   • ABSTRACTION LEVELS
   • DOCUMENT THE TESTWARE
   • GOOD DEVELOPMENT PROCESS
   • SET CLEAR GOALS
   • SET STANDARDS

Category

Failure

Experiences

If you have experienced this pattern, please add your name and a brief story of what happened, and if and how you were able to recover from it (showing links to any patterns from this wiki).

Main Page
Back to Test Automation Patterns
Back to Failure Patterns

Michael Stahl on his experiences:

I mentioned three ways in which competition may arise in a large organization, and each way should be addressed differently.

1. The case where a number of teams developed (independently) an automation framework and can’t agree on how to merge the competing solutions.
   I have encountered this situation a number of times, and witnessed the futility of trying to arrive to a “peaceful agreement” in which the teams will merge their tools to a single system. There is just too much professional pride, inertia and emotions involved.
   My conclusion is that when faced with such a situation, the only way to deal with it is to remove the arguments off the table by making a clear and uncompromising decision.
   It can be a wholesale adoption of one of the systems.
   It can be a cancelation of any additional development on any of the existing systems, merging the involved engineers under a single manager, and developing a NEW system.
   A single person must be given the authority to make binding decisions on anything that has to do with the system: scope, programming language, code repository system, architecture, etc.
   Nothing short of such drastic effort will solve the problem. As long as engineers feel that if they only come with the right argument their position will prevail, they will look for such argument. Once they realize that management made a decision and is not interested anymore in the argument, most engineers will move on and execute.
   Indeed, following this advice does mean that some of your engineers will be too frustrated to continue, and will prefer to move out of the team. I believe most of these situations can be avoided if engineers’ feel respected and not ignored. That is, while you make a decision they don’t like, make it clear – in 1/1 meetings – that you value their inputs and their work, but the situation had to be resolved, and this time it resolved differently than what they thought. But it’s not because their views were ignored or their work unappreciated; it was because there were too many strong, justifiable but opposing opinions.
   And be ready to bite the bullet and lose some talent.
2. The case in which some teams start developing a parallel automation solution because their requirements to the existing system are constantly de-prioritized.
   You can’t solve this with a better prioritization work. There will always be some requirements that get pushed out, and the team who needs this functionality will eventually look to solve the problem outside the existing system.
   The one solution I am aware of for this problem is to design the automation system with plug-in capabilities. When a system has this capability, all teams can get what they need: they just develop what they need by themselves. The benefit is that all these small efforts are controlled within one large framework; can be re-used by other teams, and conform to some development and quality standards.
   Looking at examples such as [Jenkins] and [Firefox], each of which have hundreds or even thousands of plugins, you get some confidence that this approach is a workable one. It does mean that when architecting your system (at Stage 2, hopefully), you mandate a plug-in architecture
3. In the 3rd case, teams get disillusioned with a “Stage 5” automation system, and start their own automation initiatives.
   My advice here is this: as long as these are Stage 1 tools – accept this behavior. If the new tools allow you to stop using some of the more problematic parts of the main automation system, it’s a good interim solution and will give you some breathing space to fix the main system.
   However, be very vigilant. Don’t let the tools mushroom. If possible, define some minimal coding standards, so that these tools can later be all used as plugins in a new system.
   And yes, you need to fix the main problem… redesign the current system!