Mgic Quadrant for Software Test Automation

Published: 15 November 2016 

Summary

The need to support faster time to market with higher quality is driving the demand for effective functional test automation tools. We evaluate vendors in this space to help application leaders who are modernizing software development select test automation tools that best match their needs.

Strategic Planning Assumption

By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks and open-source quality tools.

Market Definition/Description

The software test automation market offers tools, technologies, components and services that together constitute the critical elements for automated testing. It includes tools for performing static code analysis, functional testing, and load and performance testing in an automated way. This Magic Quadrant focuses on functional software test automation tools.

Functional testing is a form of testing that deals with how applications function. A functional software test automation tool enables an organization to design, develop, maintain, manage, execute and analyze automated functional tests for applications running on different platforms (including desktop, web and mobile). These tools can run automated functional tests by driving the user interface (UI) of an application (UI test automation), or interact with the application through an application programming interface (API test automation).

There are a growing number of vendors in the test automation market as technology evolves (cloud, mobile, Internet of Things [IoT]), practices change (agile, lean, DevOps) and business moves to utilize technology as part of a digital business disruption to the market. These create opportunities for new and small vendors and new offerings to automate different layers of the release pipeline. Many solutions currently don't meet the criteria of this evaluation.

We are seeing particularly strong impact from open source upon the market. Currently, these are mainly oriented toward developers, and focused on web and mobile technologies, making them somewhat specialized and not appropriate for all testing needs.

Because of these market shifts, we expect that most organizations will utilize more than one solution for the next five years.

Magic Quadrant

Figure 1. Magic Quadrant for Software Test Automation

Source: Gartner (November 2016)

Vendor Strengths and Cautions

Hewlett Packard Enterprise

Hewlett Packard Enterprise (HPE) is in the Leaders quadrant this year, based on its completeness of vision and continued strong execution globally — particularly in building strong partnerships with system integrators, including Accenture, Capgemini and Deloitte. HPE continues as the overall market share leader and is strong in relation to both richness of functionality and integration capabilities.

The test automation product set is part of the larger HPE product portfolio that includes Application Lifecycle Management (ALM), network virtualization and cloud capabilities. It consists of several complementary components covering a wide spectrum of testing scenarios and contributing roles. In addition to its core Unified Functional Testing (UFT) product for test automation engineers, HPE offers Business Process Testing (BPT) for business analysts/testers, and has addressed more-technical roles with LeanFT for agile automation engineers and developer testers. Coming from a history of more-traditional ALM-driven testing, HPE has put more focus on addressing the needs of agile development organizations, making automated testing more efficient, and helping developers and testers collaborate.

HPE offers support for native Windows applications; packaged applications including SAP, Oracle, Siebel and Salesforce; desktop and mobile browsers; web and mobile applications; and Selenium. UFT can be run on Amazon Web Services (AWS) as a cloud-based functional testing tool, and mobile tests can be run on the AWS device farm. HPE's DevOps vision continues to evolve with support for different CI systems, Git, Docker and network virtualization.

STRENGTHS

• HPE promotes a continuous testing approach with a product set for automating tests at scale that targets a wide range of personas, from technical experts to business experts.
• HPE UFT delivers a very strong set of capabilities and offers support for UI technologies, mobile, and API and web service testing. It has built-in BPT framework support for keyword-driven and scriptless testing, and bundled LeanFT functionality.
• HPE has a broad network of service and technology partners — including even its competitors — and a broad range of skilled resources to call upon.

CAUTIONS

• HPE's portfolio is comprehensive and can be perceived as too complex, especially in the early stages of investing in test automation or for simpler testing needs.
• Customer references have noted that HPE customer support and pricing could be improved, with a need for reduced response times and simplified pricing and licensing.
• While HPE has made good progress in enhancing its portfolio, Gartner continues to see a consistent pattern of competitors gaining market share directly by adding former HPE customers. The announced spinoff/merger of the majority of its software portfolio with Micro Focus could create uncertainty about future product strategy.

IBM

IBM is in the Leaders quadrant this year, based on its broad portfolio of technology products and services, and strong execution in the global market. It is well-positioned for large-scale enterprise testing organizations, combining its tool services portfolio to build effective practices with a strong partner network. IBM's vision continues to evolve as it relates to continuous testing and DevOps requirements.

The IBM Rational Test Workbench (RTW) is part of IBM's comprehensive Rational product portfolio that spans the full software development life cycle (SDLC), and consists of several test authoring and execution components. IBM has a strong focus on the technical tester role, and Rational Functional Tester offers multiple scripting options (Java, Visual Basic .NET) and a straightforward record/enhance/execute paradigm. Similarly, its solution for API testing from the Green Hat acquisition, Rational Integration Tester, is targeted at the technical tester. IBM combines its cloud offering (Bluemix), services and support for open source to address the needs of DevOps organizations.

The RTW Web UI Tester provides automated testing for HTML5-based browser applications for various browsers and OSs, and the RTW Mobile Test Edition supports testing for native, hybrid and web apps on Android and iOS devices. Rational Functional Tester automates functional tests for traditional GUI clients and the mainframe, while Rational Integration Tester enables API testing and can be combined with Rational Test Virtualization Server for service virtualization.

STRENGTHS

• IBM's test automation tools can give many enterprises a jump-start in their continuous testing efforts. They are strongest as an integrated solution, delivering a combination of products and services.
• IBM provides automated testing support for many technologies and applications, including legacy systems, client/server applications, packaged applications, and Web and mobile applications. It offers integration across the development and delivery life cycle.
• IBM is well-positioned for cloud-based testing and advanced cognitive analysis to aid in adaptive testing and decision making.

CAUTIONS

• IBM's portfolio is comprehensive and can be perceived as too complex, especially in the early stages of investing in test automation or for simpler testing needs.
• While IBM provides a powerful portfolio, it is not optimized for nontechnical users and may require additional services to set up and implement.
• IBM supports packaged application testing, but relies on partners for business process analysis and validation.

Micro Focus

Micro Focus, which has now fully incorporated Borland following its earlier acquisition, is in the Visionaries quadrant this year, based on its strong capability to support the needs of IT and line of business (LOB) users, as well as mobile testing enhancements. Its vision for automated testing is driven by its Application Lifecycle Management (ALM) heritage, but has evolved to better address cross-browser and mobile application testing use cases as well as complex end-to-end testing scenarios.

Micro Focus is primarily known for products covering different aspects of the SDLC, including requirements management, change management and software quality. Micro Focus's test automation product Silk Test enables rapid development of automated tests using a number of different automation paradigms, such as record/replay and keyword-driven testing. Micro Focus focuses on the collaboration between business and IT, and enables different user types to contribute to test automation. Silk Test includes a visual testing interface for business-focused users, as well as Java and .NET developer-focused interfaces.

Based on its history in desktop application testing, Micro Focus offers broad support for testing Windows applications built using native Windows, Java or .NET technology stacks. Web application testing is supported for desktop and mobile browsers, rich internet application technologies, and cross-browser testing leverages Selenium. Mobile applications can be tested on iOS and Android. Silk Test offers image and text recognition, and extension mechanisms for custom UI objects. Micro Focus provides integrations with existing development tools such as build and continuous integration (CI) servers, source control, or test management systems.

STRENGTHS

• Micro Focus promotes a collaborative approach to test automation by enabling different roles, such as business analysts, test automation engineers and developers, to contribute to the process of creating and maintaining automated tests.
• Micro Focus provides automated test support for many technologies and applications, including Windows desktop, mobile, web, packaged applications and green screen. It provides strong support for development processes with integration to several DevOps pipeline tools, including CI and test management.
• Micro Focus reference customers consistently report customer service and ease of use for both technical and nontechnical users as strengths

CAUTIONS

• Micro Focus Silk Test does not offer functional testing for APIs or web services, although does provide integrations to other API testing tools that support this.
• Micro Focus offers support for testing of package applications, but has no accelerators or complementary capabilities such as business process or change impact analysis.
• While Micro Focus has made good progress in integrating overlapping testing products from earlier acquisitions, its announced intent to merge with the majority of HPE's software portfolio could create uncertainty about future product strategy.

Microsoft

Microsoft is in the Challengers quadrant this year, based on strong execution globally with its development and testing offerings within the sizable .NET developer market. Microsoft's vision for test automation continues to evolve as it unifies its acquired, partner-provided and organically developed testing products to catch up in areas such as technology support and usability for nontechnical testers.

Microsoft's test automation solution is based mainly on Visual Studio and the Xamarin platform (acquired in early 2016) for test authoring, and Visual Studio Team Services, Xamarin Test Cloud and Team Foundation Server (TFS) for test execution. The primary roles Microsoft focuses on are the developer and the developer tester, and its products offer a wide range of scripting options and support for many testing frameworks. A record and playback experience as well as visual tools for managing automated tests also address the nondeveloper tester. Microsoft has put a strong focus on continuous testing and cloud services.

Microsoft offers support for native Windows applications based on the .NET stack, web (directly or using Selenium) and mobile (Android, iOS, Windows Phone), and provides Coded UI templates for testing specific targets such as the Universal Windows Platform or Windows Phone apps. Additional technology support is provided through partner plug-ins, and the IntelliTest feature generates units for .NET code.

STRENGTHS

• Microsoft is a solid choice for enterprises that have significant investment and skills in the Microsoft ecosystem: Visual Studio, .NET and Windows-based back-end systems.
• Microsoft has a strong connection to the developer audience and its offering provides strong support for DevOps practices, with tight integrations that support continuous testing and cloud scalability for test execution.
• The acquisition of Xamarin greatly enhances Microsoft's position in mobile testing, and allows .NET developers to employ their skills in the development and testing of mobile apps.

CAUTIONS

• Microsoft's test automation solution is a collection of developer-oriented tools that provides little support for automation by nondevelopers and has not generated awareness or traction among higher-level LOB buyers.
• While Microsoft supports testing for Microsoft Dynamics applications, it doesn't provide any frameworks or accelerators and offers no support for testing packaged applications.
• Customer references have had concerns about product quality and functionality, and noted that there is still room for improvement in Microsoft customer support.

Ranorex

Ranorex is in the Niche Players quadrant this year, based on its limited distribution strategy and focus on software development organizations (ISVs). Ranorex is executing well in its market and has strong products and technology, but has not extended its vision to address the broader testing requirements of large enterprises, such as business process testing, or nontechnical test team members, such as business analysts.

Ranorex's test automation product, Ranorex Studio, offers authoring of automation test suites based on the Microsoft .NET framework. It provides automated testing, data- and keyword-driven testing, and cross-platform and device testing from within a single tool. Ranorex focuses on developers and testers, providing the ability to write test code in C# or Visual Basic .NET (VB.NET), or create reusable action modules using the Ranorex Recorder facility. Overall, Ranorex has put a strong focus on developer support, and professional programmers can use an API for C# and VB.NET to enhance recorded tests.

Ranorex offers broad support for the Windows platforms and applications built using native Windows, .NET or Java technology stacks, including many popular GUI frameworks. It supports web and mobile application testing for a wide range of web development front-end toolkits, browsers (desktop and mobile), Selenium and mobile applications (iOS and Android). Visual Studio developers can use the Ranorex test automation framework to implement Coded UI tests that execute Ranorex-based automation code, using Microsoft Test Manager as part of an existing Team Foundation Server infrastructure.

STRENGTHS

• Ranorex offers strong UI object recognition support for Windows, web and mobile application technologies, and provides strong support for development with integration to several DevOps pipeline tools, including CI and test management.
• Ranorex offers a straightforward and relatively low-cost license model that supports testing for a wide range of technologies and platforms with one license.
• Ranorex reference customers consistently report product functionality and quality, time to become successful, customer service and ease of use for both technical and nontechnical users as strengths.

CAUTIONS

• UI object recognition is very powerful but not always straightforward — for example, developing stable, highly parameterized scripts can get complicated. Ranorex does provide scripting advice to help address some of these challenges, however.
• Although Ranorex offers support for recognizing many UI elements of packaged applications, it offers no accelerators or complementary capabilities such as process or impact analysis.
• Ranorex has a number of distribution and service partners, but lacks strategic partnerships with the large system integrators that could help it make inroads into large enterprises.

SmartBear

SmartBear is in the Visionaries quadrant this year, based on its strong capability to support the needs of IT organizations, and investments in extending its test automation portfolio to address additional use cases. Its vision for automated testing has evolved to better address cross-browser and mobile application testing, API testing and Internet of Things (IoT) testing, as well as to reuse test assets in complex end-to-end testing scenarios.

As part of its wider portfolio of testing products, SmartBear offers test automation products for UI testing for desktop, web and mobile applications, as well as API testing and cloud-based cross-browser testing. SmartBear's TestComplete Platform enables QA engineers to develop automated tests using both keyword-driven and programmatic development styles. Its TestLeft product enables developers to contribute to test automation via an integration with Visual Studio or RAD Studio. Overall, SmartBear focuses on improving collaboration between development and test teams, and engaging in open-source communities (e.g., SoapUI and Swagger).

TestComplete offers broad support for Windows applications containing standard Windows or popular GUI framework controls, or built with .NET and Java technology stacks. Web and mobile application testing includes support for a wide range of web development front-end toolkits, desktop and mobile browsers, Selenium, and iOS and Android OSs. SmartBear also supports API testing and service virtualization, and integrates with build and CI servers, defect tracking tools, source control tools, and test management systems.

STRENGTHS

• SmartBear is known for providing well-rounded capabilities for functional testing of desktop, web and mobile applications, as well as APIs and web services. It enables different roles to contribute to the process of creating and maintaining automated tests.
• SmartBear's acquisition of CrossBrowserTesting has strengthened its capabilities in responsive web design testing on multiple browsers. It also enables support for Selenium and Appium, which can be combined with automated tests created using TestComplete and run as an integrated regression suite.
• SmartBear has a strong community that supports customers and helps them to expand and customize products for their specific needs. It provides a plug-in framework that customers can use to extend TestComplete and share it with other users in the community.

CAUTIONS

• SmartBear's support for automated testing of packaged applications is limited to Oracle E-Business Suite, and it offers no complementary capabilities such as business process or change impact analysis.
• While the process of product upgrade is straightforward, some SmartBear reference customers have reported occasional product quality and instability issues.
• SmartBear is strongest in North America and EMEA, but relies on partners for support in other regions.

TestPlant

TestPlant is in the Visionaries quadrant this year, based on its strong capability to support the needs of IT and LOB users with an easy-to-use full test automation suite. Its vision for automated testing is driven by a technology-agnostic, noninvasive, cross-platform approach that is well-suited for complex end-to-end testing scenarios.

TestPlant's test automation offering is part of a comprehensive testing product portfolio covering functional testing, performance testing, network emulation and test management. Using an image-based approach to GUI testing, TestPlant's eggPlant Functional product enables rapid development of technology-agnostic, cross-platform test automation using different testing paradigms. TestPlant focuses on enabling nontechnical users to develop automated tests using both keyword-driven and programmatic development styles. Developers can take advantage of the eggDrive APIs for Java and .NET to drive eggPlant Functional.

TestPlant's test automation products offer a platform-independent approach and don't provide support for specific UI technologies or platforms. Its image-based object recognition enables automated testing of desktop applications, mobile devices and mainframes, as well as point of sale (POS) systems and any other device that has a GUI. TestPlant provides integrations with CI and ALM tools, and domain-specific testing tools (e.g., for vehicle Controller Area Network [CAN] bus systems, retail payment systems, KVM switches and so on).

STRENGTHS

• TestPlant's functional testing solution has shown a strong ability to support a wide range of technologies, OSs, platforms and devices. It has extended capabilities to support nonstandard UIs and custom devices from industry-specific applications.
• TestPlant provides users with the ability to execute testing on multiple devices across different platforms using a single script. Reference customers have cited ease of use as a strong point, resulting in improved productivity.
• Reference customers have consistently noted exemplary customer service support rendered by TestPlant, and also scored it highly on product functionality and capabilities.

CAUTIONS

• TestPlant does not offer functional testing of API or web services, but can recognize and call APIs at GUI-level test execution. It does not focus its offering on testing packaged applications, although is used by some customers for this purpose.
• TestPlant's technology relies on an image recognition paradigm that is easy to use and enables rapid support for a wide range of technologies and platforms, but reference customers have noted that maintenance can be challenging at times.
• TestPlant is strongest in North America and EMEA, but relies on partners for support in other regions. However, it is looking to improve strategic partnerships with the large system integrators that could help it make inroads into large enterprises.

Tricentis

Tricentis is in the Leaders quadrant this year, based on completeness of offering and its continued strong execution globally — particularly in building partnerships with system integrators, including Accenture, Infosys and Wipro. Tricentis's vision continues to be very strong, relating to continuous testing and enabling manual testers to become test automation experts.

Tricentis's test automation capabilities are part of the Tricentis Tosca Testsuite, a comprehensive solution including requirements, test case design, test management and test automation functionality. The concept of model-based test automation links the solution's different components, and enables rapid development of automated tests using a number of different automation paradigms and testing styles. The Tricentis Tosca Testsuite is a script-free tool that uses table views and graphical representations of test cases and process flows. Tricentis has a strong focus on the business analyst and manual tester roles, and on making them effective contributors to test automation. Overall, Tricentis takes a holistic approach to making automated testing more efficient, with risk-based test case design and planning.

Different editions of the Tricentis Tosca Testsuite offer different combinations of UI testing for desktop, web, mobile and modern apps. The Premium Bundle includes all supported technologies. Tricentis offers support for native Windows applications; packaged applications including SAP, Oracle, Salesforce, desktop and mobile browsers; web and mobile applications; and Selenium. It supports API testing and service virtualization, and can run tests both on-premises or in the cloud, using Tosca as a Service (TaaS).

STRENGTHS

• Tricentis offers a strong combination of risk-based test case design, model-based test automation, integrated test data management and strong analytics well-suited for business analysts, manual testers and QA professionals.
• The Tricentis Tosca Testsuite delivers a very strong set of capabilities for testing UI technologies, mobile, APIs and web services, and has very strong support for packaged applications, including solution accelerators.
• Tricentis has made good progress in building strong relationships with service and technology partners, including for joint marketing activities, training and certification of a significant number of resources.

CAUTIONS

• Tricentis reference customers have noted that the time and effort to become successful and productive in automating test cases could be improved.
• Tricentis is strongest in EMEA and APAC, and does not yet have the same footprint in North America. However, it has addressed this issue and has won some large deals in the region.
• While Tricentis supports agile development and continuous delivery (CD), its script-free approach is different from what most agile developers expect.

Worksoft

Worksoft is in the Challengers quadrant this year, based on strong execution globally and strong partnerships with system integrators. Worksoft is executing well in its market, which is specifically business process testing for packaged enterprise applications. Worksoft has intentionally not pursued support for code-level testing or service virtualization, which makes it less applicable for highly custom-developed applications.

Worksoft's test automation solution consists of Worksoft Analyze for capturing business process workflows, and Worksoft Certify for refactoring the captured test steps and executing tests. The primary roles of focus are business analysts and subject matter experts, who provide the input from which tests are generated, as well as QA professionals and testers, who further develop and maintain the test automation assets. The Worksoft tools use a script-free approach with graphical representations of process flows and table views. Overall, Worksoft has a strong and unique focus on analyzing, documenting and testing enterprise business processes that are implemented in packaged applications, as well as on business process automation.

Worksoft Certify supports testing of a wide range of packaged applications, including SAP, Oracle, Salesforce and Workday, as well as desktop, web and mobile (through devices hosted by Perfecto and Keynote by Dynatrace partnerships, which are accessed via APIs and entirely written in Worksoft Certify). Worksoft offers integrations with test management/application life cycle management products including HP ALM (Quality Center), SAP Solution Manager and IBM Rational Quality Manager, as well as defect and issue management systems including Atlassian Jira, ServiceNow and Syergex Serena TeamTrack.

STRENGTHS

• Worksoft offers a full range of script-free tools, including business process analysis and documentation and change impact analysis, to accelerate testing for ERP business users who need to keep up with frequent upgrades and enhancements.
• Worksoft is noted for having a single solution for packaged applications as well as custom-developed solutions for web and mobile via a code-free approach.
• Worksoft has strong relationships with service and technology partners — including even its competitors, which use Worksoft's offering as part of their solutions and act as influencers for additional opportunities.

CAUTIONS

• Worksoft reference customers have noted that the time and effort to become successful and productive in automating test cases could be improved.
• Worksoft provides its own development environment, but intentionally does not offer support for IDEs, scripting or programming languages, or source control tools, as this is not its core market. This makes it less attractive for developers seeking to script or program.
• Although Worksoft has integration capabilities with third-party application development life cycle management and DevOps solutions, it lacks integration with code development tools. This is an area Worksoft has chosen not to address.

Vendors Added and Dropped

We review and adjust our inclusion criteria for Magic Quadrants as markets change. As a result of these adjustments, the mix of vendors in any Magic Quadrant may change over time. A vendor's appearance in a Magic Quadrant one year and not the next does not necessarily indicate that we have changed our opinion of that vendor. It may be a reflection of a change in the market and, therefore, changed evaluation criteria, or of a change of focus by that vendor.

Added

Micro Focus (changed from Borland, which it has now fully incorporated following its acquisition)

Microsoft

Dropped

Automation Anywhere

Borland (see Micro Focus above)

Oracle

Original Software

Progress

Honorable Mentions

The following vendors did not meet the inclusion criteria, but are credible alternatives:

Seapine Software (provides tools to automate functional and regression testing of web, Windows and Java applications)

SmarteSoft (offers scriptless automated testing and healthcare-specific features for automated testing of leading hospital information systems and electronic health record environments)

Zaptest (offers test automation for mobile, web and desktop applications, and allows testers to execute the same script developed for one technology platform of the same technology on any other platform)

Inclusion and Exclusion Criteria

Vendors in this year's Magic Quadrant met the following criteria:

• Provided the ability to create, develop, manage and execute automated functional tests on the Windows platform. Provided the ability to execute tests on at least the Android and iOS mobile platforms (additional platform test execution support, such as Mac OS or Linux, is desirable).
• Provided the ability to test at least three of the following: responsive web applications, mobile applications, package applications, API/web services (in addition to native Windows desktop applications).
• Software revenue in 2015 of more than $10 million from their software test automation product license.
• Global market representation with significant presence in all major regional markets (Americas, EMEA and APAC).
• Added at least 10 new paying enterprise customers in at least two regions in 2015.

Vendors were excluded on the following terms:

• Offered only test execution platforms, without a tool to create, develop or manage automated tests.
• Offered mobile-only testing (see Note 1 and "Market Guide for Mobile App Test Automation Tools" ).
• Offered commercial off-the-shelf (COTS)-only testing tools (see Note 2).
• Only sell their software coupled with development/professional services where the tool is used exclusively by company consultants.
• Do not sell a commercial enterprise offering (that is, only offer the solution as open-source software). While we are seeing strong uptake of many open-source components, we have not included open-source solutions because of the revenue inclusion requirement.

Evaluation Criteria

Ability to Execute

Gartner analysts evaluate technology providers on the quality and efficacy of the processes, systems, methods or procedures that enable IT provider performance to be competitive, efficient and effective, and to positively impact revenue, retention and reputation. Ultimately, technology providers are judged on their ability and success in capitalizing on their vision.

The Ability to Execute in a consistent fashion by providing a combination of technology with very clear market positioning is critical. Better technology is important, but generally opens only short-term windows until the incumbents acquire it or catch up. Core test automation functionality is fast becoming a commodity, and vendors increasingly compete by adding value on top of the underlying frameworks and tools. Leaders with the Ability to Execute deliver not only a strong product, but also a comprehensive offering that includes services, support, strong communities and partner ecosystems, which collectively will enable successful utilization of these products.

The leading vendors also provide good support for emerging practices and integrations. Time-to-market pressures are driving adoption of agile and DevOps practices. The automated execution of tests as part of a CD process is replacing the more traditional ALM-driven execution of tests, and agile teams tend to have very different requirements than those of traditional testing centers of excellence. As DevOps principles gain significant traction in enterprises, testing is no longer a stage in a DevOps delivery but an integral DevOps activity that exists in various forms through all areas. The ability to quickly enable different roles to effectively contribute to building maintainable test automation is critical, and we have rated customer experience as "high."

• Product or Service: We are looking for breadth and depth of products and features, including test design and creation; test maintenance and execution; and test management, integration and automated testing, as part of digital-business-driven enterprise agile and DevOps initiatives.
• Overall Viability: We are looking for R&D spend and resources, growth of software test automation business, and financial profitability or funding/capitalization.
• Sales Execution/Pricing: We are looking for broad sales reach across geographies and industries, effectiveness of sales — such as long/short sales cycles — and simplicity of pricing models.
• Market Responsiveness and Track Record: We are looking for how quickly products are released and adopted, and how test automation capabilities are supported both organically and through partnerships.
• Marketing Execution: We are looking for general awareness of the vendor in the market, as exhibited in customer interactions, presence at events and on social media, perception across IT and LOBs, and how easily buyers understand its differentiators.
• Customer Experience: We are looking for the ability to meet and exceed customer expectations in achieving increased levels of automation and maintaining that automation, ease of onboarding and training for development, and an increase in clients' overall testing maturity.
• Operations: We are looking for the development of effective materials to drive successful use of products, stability in leadership vision and strength of customer service.

Table 1.   Ability to Execute Evaluation Criteria

Evaluation Criteria	Weighting
Product or Service	High
Overall Viability	Medium
Sales Execution/Pricing	Low
Market Responsiveness/Record	Medium
Marketing Execution	Medium
Customer Experience	High
Operations	Medium

Source: Gartner (November 2016)

Completeness of Vision

Gartner analysts evaluate technology providers on their ability to convincingly articulate logical statements about current and future market direction, innovation, customer needs, and competitive forces, and how well these map to the Gartner position. Ultimately, technology providers are rated on their understanding of how market forces can be exploited to create opportunity for them.

Leaders having a Completeness of Vision are strong in their ability to reach the market via a compelling message and in delivering a solid solution globally. An increasing number of teams are globally distributed, and the majority of organizations make use of a mixture of IT, LOB users and contractors. The ability to drive success in creating and maintaining test automation for teams of individuals with different skill sets is critical.

As enterprises engage in multiyear transformations involving agile, DevOps and bimodal practices, the ability of a vendor to support test automation that consistently creates value along the way and drives positive change will make them a preferred partner. The key messages now are around productivity and the ability to aid in technology and process transformation.

We have rated innovation as "high" and looked specifically at new and differentiating capabilities in these areas:

• Technology — for example, visual testing
• Continuous testing — for example, enabling more-frequent releases through support of modern test practices
• Support for different roles and skill sets — for example, developer tester, test automation engineer, business analyst

Leaders have a complete vision or an ability to support everything that a quality team will need. They support leading technologies and architectures, with a clear understanding of the shift this introduces to applications and the complexity involved in testing those applications.

• Market understanding — We are looking for an understanding of how to enable customer success with automation and deal with a changing competitive business environment, as well as how to address the needs of IT and LOBs.
• Marketing strategy — We are looking for strong brand recognition, thought-leading product messaging and outreach programs that cut through a diverse testing market.
• Sales strategy — We are looking for a strong go-to-market strategy focused on selling test automation to enterprise IT, LOBs and agile developers.
• Offering (product) strategy — We are looking for a strong understanding of enterprise needs across the SDLC, and a coherent solution to address test design and creation; test maintenance and execution; and test management, integration and automated testing, as part of digital-business-driven enterprise agile and DevOps initiatives.
• Business model — We are looking for product revenue growth, ease of doing business with customers, and a strong partner ecosystem amplifying the vendor's go-to-market strategy.
• Vertical/industry strategy — We are looking for differentiating capabilities built for specific industries, vertical-specific accelerators, and a focused go-to-market approach for any specific industries.
• Innovation — We are looking for technology advancements in areas such as DevOps, IoT, wearables and omnichannel support, and scriptless automation.
• Geographic strategy — We are looking for diverse customer deployments across geographies, awareness within geographies across the globe, and in-country vendor presence.

Table 2.   Completeness of Vision Evaluation Criteria

Evaluation Criteria	Weighting
Market Understanding	High
Marketing Strategy	Medium
Sales Strategy	Medium
Offering (Product) Strategy	High
Business Model	Medium
Vertical/Industry Strategy	Low
Innovation	High
Geographic Strategy	Medium

Source: Gartner (November 2016)

Quadrant Descriptions

Leaders

Leaders have established strong market positions, as shown by breadth of adoption, global deployments and integration of other products. They provide products that are a strong functional match to general market requirements, have been among the most successful in building a loyal customer base, and have a relatively high viability rating due to strong revenue growth and/or high market share. Leaders also show evidence of superior vision and execution for emerging and anticipated market requirements, as well as a consistent track record of innovation.

They also have a good vision of the needs of the enterprise, support for open source and standards, a solid understanding of IT requirements, and scalable channels and partnerships. They generally have breadth across a significant number of supported technologies and platforms, and deliver market-leading functionality in one or more functional areas. Leaders must provide solutions that are easy to purchase, use, manage and upgrade, and which can connect to a range of complementary systems — from both the same vendor and third parties.

Challengers

Challengers have a high market reach and large deployments. Vendors in this quadrant typically have strong execution capabilities, as evidenced by financial resources, and a significant sales and brand presence garnered from the company as a whole, if not directly from its test automation-related activities. Challengers have executed well in a specific use case or market, but tend to have limited breadth or market penetration across the wider spectrum of use cases. These vendors tend to have well-established partnerships and a solid global presence.

In general, Challengers may have been innovators at one point, but are not seen as driving the application development market. They have a strong presence in a portion of the market, but have not yet created a strong presence in the entire market.

Visionaries

Visionaries have a compelling vision of products and the market's future, as well as the technical direction (and necessary resources) to take them there. However, they have not yet demonstrated that vision in one or more of the following areas: history of execution, revenue, size of client base, diversity of solutions or strong financial results. Vendors in this quadrant have a strong vision in a specific category that potentially disrupts the market through innovation or support for new technologies, but products may not be as mature or well-suited to support global enterprises.

Niche Players

Niche Players have solutions catering to specific audiences or with limited use-case support today. They are not as strong in one or more of the following criteria: product breadth/completeness or focus, geography or number of customers. In addition, vendors in this quadrant may have a much more limited ability to invest in the necessary functional as well as organizational capabilities to expand beyond their current focus. Nevertheless, for specific scenarios, an offering from a Niche Player may represent the optimal choice. In some cases, these vendors may be more mature and able to support customers than some of the Visionary providers.

Context

As organizations move toward digitization of their businesses, development teams are looking for ways to optimize the delivery of applications to the point of a continuous flow of releases into production. Automation has become an indispensable element in these efforts, and software test automation tools have seen a renewed focus as part of a larger tool chain that enables DevOps. While demand for faster delivery of applications rises steeply, the test matrix grows many times faster as delivery channels, delivery methods and client endpoints get ever more varied. Development and QA in organizations simply cannot keep pace due to a heavy reliance on manual testing processes, deficiency of skills, insufficient resources, and an inability to scale the technology and processes. These challenges are what software test automation is designed to address.

Developing bimodal capabilities will be key for organizations to succeed in digital transformation, and Mode 1 and Mode 2 teams will have different needs. In Mode 2 operations, where organizations may be targeting a high cadence for releases, automation is imperative. This automation will start with static code analysis and unit tests, but will also include functional test automation at the UI and API layer, as well as performance tests that run with every build and are kicked off via CI tools such as Jenkins, CircleCI, Travis CI or Bamboo. Tests must run efficiently and without human interaction, and should be able to report back into a dashboard. Such dashboards may be part of the CI tool, part of an agile application development life cycle management environment (e.g., CA Agile Requirements Designer, VersionOne or CollabNet), or a stand-alone quality dashboard such as SonarQube. The key is aiding a team in always knowing what the state of the build is.

For Mode 1 operations, the importance to test faster and more often is also growing — with packaged applications moving to more-frequent updates and the increased integration between internal applications and SaaS solutions. Automated testing is one of the agile development practices that is still more commonly used in Mode 2 development, but that can be applied in a Mode 1 context (see "Apply These Seven Lessons From Agile to Mode 1 Development" ). The ability to ensure, efficiently and via an automated regression suite, that the "latest update" hasn't broken systems reduces overall costs and can boost confidence.

IT organizations struggle to move from the tactical task of testing software as part of a distinct phase or activity — and often done manually — to a more automated form of QA that essentially requires no human intervention. The cost of creating and maintaining automation is often high compared with the cost of performing the test manually. However, a number of drivers are helping to bring more value to automation, including the ability to run the same tests on a number of different platforms, devices or configurations. Sets of automated tests that are specifically designed to assess the business risks associated with a release candidate can be executed on demand in the context of stable, productionlike test environments. Automation isn't just a matter of being able to quickly create a set of automated test scripts; it is about being able to create a set of effective tests that can be maintained with minimal effort and executed without any human intervention. Ultimately, tests must demonstrate that applications not only work, but satisfy real business requirements.

After reviewing this research, application and IT leaders need to take the following actions:

• If you don't already use a solution for automated testing: Utilize this Magic Quadrant to understand the market as well as evaluate tools that fit your required use cases and the technical skills of the individuals who will be engaged in automation efforts. A test automation tool will support a more repeatable and consistent testing approach across projects, and enable more scalable testing as the number of test targets increases.
• If you already use a test automation solution (either listed in this Magic Quadrant or not) and are thinking of switching providers: Consider whether it may be better to supplement the missing functionality by adding complementary tools — for example, open-source frameworks or plug-ins that provide the needed functionality. The cost of switching from one test automation solution to another could be high, depending on the amount of custom integration to your environment and customized script logic. Make sure additional test automation tools integrate with your existing solution.
• If you use a test automation tool that no longer meets your needs or that you feel may not be viable: Start evaluating other platforms — not just on current needs, but also on future application development requirements 12 to 18 months out, which may include additional technologies such as wearables or other IoT objects. You may want to pick a platform that offers similar technologies or requires similar skill sets (such as scripting or programming language support or visual paradigms), but don't lock yourself into these existing needs if your application testing requirements have outgrown them.
• If you use a test automation solution and are happy with it: Assess its capabilities and roadmap features at least every six to 12 months to make sure that it continues to align with your organization's expectations and plans.

Because of the diversity of technologies to test, and as organizations become bimodal, teams should also expect to have more than one automation tool in place. Note that we have Market Guide documents already available for:

• Mobile test automation (see "Market Guide for Mobile App Test Automation Tools" )
• API testing and service virtualization (see "Market Guide for API Testing and Service Virtualization" )
• Test management (see "Market Guide for Test Management Tools" )
• Performance testing (see "Market Guide for Performance Testing" )

Users should evaluate all of these as they evaluate their overall testing tool needs.

Market Overview

The success of digital business transformation strategies is dependent on a much more comprehensive approach to quality than just answering the question, "does it work?" Applications must excel in overall user experience, quality of service (QoS), availability, performance, frequency of updates and bug fixes. Modern, omnichannel enterprise applications that implement complex business scenarios are built using an increasingly complex technology stack that is creating new challenges for development and testing organizations. They must find ways to achieve a higher rate of automation to keep up with the platform spread and pace of change demanded by the business. As companies seek to deliver more-flexible and richer internet experiences, the increasing need to support large sets of mobile devices (including wearables) and incorporate IoT devices increases the diversity and complexity of test environments and test scenarios by orders of magnitude. This development is making investment in automation more cost-effective, because the same tests can be run on multiple browsers, browser versions, desktop and mobile OSs, mobile devices and so on.

The growth in DevOps is also creating a strong demand for automation technologies that we expect to last through the next five years, as most organizations currently have very low automation rates. CD practices rely on a fully automated delivery pipeline, including automated functional testing (as well as code quality, security and performance tests).

Thus, the shift toward web and mobile applications, agile development and DevOps practices, and the emergence of new players and open-source offerings continue to disrupt the market. Growing competition is pushing all vendors to improve and has created positive price pressure. We believe that the market will continue to go through a consolidation cycle during the next five years, with acquisitions to accelerate growth and shore up aging portfolios. There is potential for two or three more vendors to move into the Leaders quadrant, driven by the shift to mobile and cloud applications.

During the course of producing this research, Micro Focus and HPE announced a spinoff/merger of the majority of HPE's software portfolio with Micro Focus (see "HPE's Spinoff/Merge of Its Software Businesses to Micro Focus May Create Significant Challenges for Users" ). This is broader than the testing products, but we note that the current market is very dynamic from both the standpoint of private equity as well as commercial providers expanding portfolios to deal with CD. CA Technologies also announced the acquisition of BlazeMeter, and we expect that this type of activity will accelerate as vendors look to position themselves in the digital business marketplace, consolidate market share or build a solid maintenance revenue pipeline. Users should be prepared for this as they negotiate contracts with vendors, including change in control policies that will protect them in their use and the costs of software.

Open source is having a strong effect on the market, and has been well-adopted by organizations practicing agile development and making use of CD practices. They are replacing the more traditional ALM-driven execution of tests with the automated execution of tests, as part of a CD process. Testing is evolving to a "bring-your-own-tools" mentality; and rather than use a monolithic tool, developers are using a combination of open-source and commercial tools to perform the various testing activities:

• Selenium has emerged as the de facto standard for web testing, and a large ecosystem has developed around it. Several vendors sponsor development (e.g., Sauce Labs, BrowserStack, Experitest, CrossBrowserTesting [a SmartBear company]), and many are providing value on top of the core execution framework — for example, by supporting continuous testing and enabling IT organizations to execute a set of tests specifically designed to assess the business risks associated with a release candidate.
• Appium, backed by Sauce Labs, is also gaining a lot of momentum as one of the most popular open-source, cross-platform mobile automation frameworks for executing tests on iOS and Android, as well as Windows desktop.
• BDD frameworks such as Cucumber, JBehave and Behat along with the Gherkin language are getting adopted by many agile development organizations. Test automation vendors have started to provide support for such frameworks.
• A number of other open-source tools and frameworks have also gained momentum, including Geb, SoapUI, Sahi, Watir, Protractor, Bugzilla and Apache JMeter, and the number is growing.

This is increasingly dividing the market. On the one hand, there are many developer-centric tools that integrate well with the overall tool stack and a code-based approach, but are less-suited for nondevelopers to utilize. At the same time, developers are generally skeptical of tools that either hide or remove the coding. We expect that the majority of organizations will find themselves using a number of tools from different providers (or open-source projects) rather than having a single vendor to solve all problems.

This potentially disruptive situation typically leads application and IT leaders to evaluate and adopt test automation tools for the purpose of accelerating and scaling application development and testing activities.

Test automation tools serve four critical functions in this regard:

1. Offer a test case development facility or IDE that increases productivity for cross-platform (OSs, devices), multichannel (web, apps, wearables) and thick client (legacy applications, ERP, desktop) test automation development.
2. Enable integration into multiple test management tools, ALM platforms and quality dashboards, creating the ability to leverage detailed test result information for advanced quality-based decision making.
3. Support continuous testing and execution of tests in the context of stable, productionlike test environments.
4. Support popular third-party solutions such as multiple scripting languages, BDD frameworks, test automation frameworks, and other open-source quality tools. Provide additional functionality on top of their core capabilities.

As automated testing tools become an integral part of a DevOps tool chain, test automation solutions are continuing to evolve rapidly beyond being just silos for the development of automated tests by test automation engineers. Indeed, nearly all of the test automation vendors listed this year enable multiple user types to contribute to building automated tests for multiple test targets, such as web, mobile, desktop or packaged applications, and foster collaboration in the greater team in the same tool.

To achieve this metamorphosis, three important principles must be addressed in test automation tools:

1. Decoupling test development methodology and technology — A test automation tool offers a comprehensive approach to test automation development, but the use cases across an enterprise are so diverse that a single test development front end cannot effectively address them all. The test automation tool must therefore make its automation services consumable by different front ends, frameworks and IDEs that are more suitable for specific app use cases and user roles.
2. Embracing open source and standards — Over the last decade we have seen a constant increase in the adoption of open-source testing frameworks and tools, and more open-source solutions continue to enter the market as additional needs drive the requirement for yet more flexibility. If a test automation solution does not embrace standards-based technologies (such as HTML5 or JavaScript) and open-source software (such as Selenium, Appium or SoapUI), it may not be able to keep pace with changes in the market or foster broad community support for the platform. The use of proprietary technology also causes vendor lock-in and, therefore, difficulty in migrating off a test automation solution.
3. Enabling self-service development — Test automation solutions have traditionally served professional test automation engineers, but test automation development is becoming democratized with more tools adopting visual or model-driven paradigms. This trend is influenced by the multichannel web testing market, where simplicity and speed are the mantras and citizen testers (such as business analysts, LOB professionals and marketing professionals) are the practitioners. Test automation development is increasingly initiated and supported by constituents outside IT, so test automation solutions must adapt to enable this self-service mode of development.

Due to this Magic Quadrant inclusion criteria and methodology, many capable providers are not included in this study. However, they may be a better fit for specific automated testing needs, depending on the type of application, industry, user skill set, development methodology or other factors. Clients are advised to speak to the authors of this Magic Quadrant to narrow down a shortlist of best-fit vendors in this study as well as to discuss providers not covered here.

In addition, there are many other functional components of a complete testing solution. Some notable areas are:

• Cross-browser testing
• Test planning and management (see "Market Guide for Test Management Tools" )
• Load and performance testing (see "Market Guide for Performance Testing" )
• API testing and service virtualization (see "Market Guide for API Testing and Service Virtualization" )
• Automation
• Mobile (see "Market Guide for Mobile App Test Automation Tools" )
• Test data and test lab
• Packaged applications or market-focused software testing
• Visual testing

Market Outlook

This year, three of the nine vendors covered reside in the Leaders quadrant. However, the reality is that core automation capabilities have become a commodity, and there is much greater parity than disparity in this market. For many organizations, product selection will be affected by the value vendors are adding on top of basic automation functionality, as well as existing relationships, support for specific technologies or solution spaces, testing service providers, or cost reduction. We expect that most organizations will have more than one automation tool provider, and that this is unlikely to change during the next five years.

For the use cases we have outlined, many of the solutions are still maturing. This is to be expected, given the dynamic nature of the mobile and web market, the evolution in COTS software with increasing SaaS functionality, and a strong move to DevOps.

Acronym Key and Glossary Terms

API	application programming interface
BDD	behavior-driven development
CI	continuous integration
IDE	integrated development environment

Evidence

The Magic Quadrant is a reflection of a broad-based research effort involving:

• Over 500 inquiries with Gartner clients inquiring about test automation tools during the past 12 months.
• Many in-person discussions and other interactions with the vendors within this Magic Quadrant.
• A detailed vendor survey requiring responses to more than 200 questions.
• Gartner conducted a survey of organizations using online tools from July to August 2016. The survey participants were customer references nominated by each of the vendors in this Magic Quadrant. These surveyed customers were asked 50 questions about their experiences with their vendors and solutions. The results were used in support of the assessment of the software test automation market. We obtained 46 full responses representing companies headquartered across several different geographic regions.
• A live product demonstration from each of the nine participating Magic Quadrant vendors, where each was requested to provide insight into their ability to support specific functions.

Note 1 
Mobile Application Testing

The mobile testing market is very dynamic, evolving as core capabilities (automation, device clouds) grow. We expect continued market consolidation, driven by acquisitions, pressure from open-source and mobile-platform-provided tools, as well as rapidly changing market needs. The majority of the vendors covered in this Magic Quadrant support mobile application testing, as this capability becomes more mainstream. This is via licensed technology or is vendor-native.

Also, many of the vendors covered in the "Magic Quadrant for Mobile App Development Platforms" provide some level of mobile testing capability. However, we note that the mobile-focused vendors offer significant advantages in many cases, including breadth of device support, speed of support of OS upgrades (new gestures, for example), and support for provisioning and management of device clouds.

We have not included tools that are mobile-only in their focus; we currently cover this area in our "Market Guide for Mobile App Test Automation Tools."There is also a large variety of options from the open-source community — specifically, Appium is gaining a lot of momentum.

Note 2 
Commercial Off-the-Shelf (COTS) Testing

One of our core use cases in this Magic Quadrant is the ability to test packaged applications. In general, we find that most vendors "support" testing of packaged application software to the extent that they can drive the UI of the front end — that is, if the client front end is a native Windows or web application, it is "just another application" that can be automated in the same way as others built on that technology. However, this ignores that packaged application testing scenarios are different to most custom-made software. Organizations are more focused on validation of business process, integration points and performance. We have not included tools that support only COTS applications in this research.

Support for packaged applications from commercial vendors is also limited. The majority of solutions focus on the ERP and/or CRM applications of SAP and/or Oracle. There is now a growing set of solutions for SaaS offerings such as Salesforce, but coverage is very uneven. Many testing service providers also have packaged application-specific offerings, and some of these include vertical-market-specific packaged applications (such as Infosys Finacle).

The best-focused solutions provide additional acceleration, including management of test data, connection to change management (to automate selection of tests that need to be run) and change impact analysis.

Evaluation Criteria Definitions

Ability to Execute

Product/Service: Core goods and services offered by the vendor for the defined market. This includes current product/service capabilities, quality, feature sets, skills and so on, whether offered natively or through OEM agreements/partnerships as defined in the market definition and detailed in the subcriteria.

Overall Viability: Viability includes an assessment of the overall organization's financial health, the financial and practical success of the business unit, and the likelihood that the individual business unit will continue investing in the product, will continue offering the product and will advance the state of the art within the organization's portfolio of products.

Sales Execution/Pricing: The vendor's capabilities in all presales activities and the structure that supports them. This includes deal management, pricing and negotiation, presales support, and the overall effectiveness of the sales channel.

Market Responsiveness/Record: Ability to respond, change direction, be flexible and achieve competitive success as opportunities develop, competitors act, customer needs evolve and market dynamics change. This criterion also considers the vendor's history of responsiveness.

Marketing Execution: The clarity, quality, creativity and efficacy of programs designed to deliver the organization's message to influence the market, promote the brand and business, increase awareness of the products, and establish a positive identification with the product/brand and organization in the minds of buyers. This "mind share" can be driven by a combination of publicity, promotional initiatives, thought leadership, word of mouth and sales activities.

Customer Experience: Relationships, products and services/programs that enable clients to be successful with the products evaluated. Specifically, this includes the ways customers receive technical support or account support. This can also include ancillary tools, customer support programs (and the quality thereof), availability of user groups, service-level agreements and so on.

Operations: The ability of the organization to meet its goals and commitments. Factors include the quality of the organizational structure, including skills, experiences, programs, systems and other vehicles that enable the organization to operate effectively and efficiently on an ongoing basis.

Completeness of Vision

Market Understanding: Ability of the vendor to understand buyers' wants and needs and to translate those into products and services. Vendors that show the highest degree of vision listen to and understand buyers' wants and needs, and can shape or enhance those with their added vision.

Marketing Strategy: A clear, differentiated set of messages consistently communicated throughout the organization and externalized through the website, advertising, customer programs and positioning statements.

Sales Strategy: The strategy for selling products that uses the appropriate network of direct and indirect sales, marketing, service, and communication affiliates that extend the scope and depth of market reach, skills, expertise, technologies, services and the customer base.

Offering (Product) Strategy: The vendor's approach to product development and delivery that emphasizes differentiation, functionality, methodology and feature sets as they map to current and future requirements.

Business Model: The soundness and logic of the vendor's underlying business proposition.

Vertical/Industry Strategy: The vendor's strategy to direct resources, skills and offerings to meet the specific needs of individual market segments, including vertical markets.

Innovation: Direct, related, complementary and synergistic layouts of resources, expertise or capital for investment, consolidation, defensive or pre-emptive purposes.

Geographic Strategy: The vendor's strategy to direct resources, skills and offerings to meet the specific needs of geographies outside the "home" or native geography, either directly or through partners, channels and subsidiaries as appropriate for that geography and marke