The Power of Automated Testing and Test Management with SmartBear and Zephyr
Make sure you get the highest level of insight into your testing practices.
·
Join the DZone community and get the full member experience.
Join For FreeDr. Milan Verma, Zephyr's Lead Client Services Engineer, and Greg Hanson, Senior Director of Global Sales Engineering for SmartBear, recently gave a presentation and answered some essential questions on automated testing and test management with SmartBear and Zephyr. "The focus of our talk today is test automation and the power it brings to your testing journey," Greg said, by way of introduction.
"Once we help you figure test automation, we're going to talk about how to combine it with your test management platform to ensure that you get the highest level of insight into your testing practices. Then, I'll pass the mic to Milan and he will give you an in-tool demonstration of Zephyr and TestComplete, combining those two worlds of test management and test automation."
SmartBear Overview
Greg noted that SmartBear was founded in 2009 and currently supports more than six and a half million software professionals and over 22,000 companies in 194 countries, on almost every continent. "We do have some deals with penguins in Antarctica that we're working on, which may take a little more time," he joked.
Beginning with a high-level overview of SmartBear product offerings, Greg explained that SmartBear has products that stretch the entire software development life cycle, from development through testing and operations — for both front-end and back-end applications.
"From the development side, we have collaboration tools for traditional and API development to allow developers, testers and product teams to really communicate as the code is being developed to help find those defects earlier in the process."
"Moving into the testing phase, we have test automation tools for developers, for people brand new to testing, and for sophisticated automation engineers--again for both the front-end and APIs in the back end."
"All the way through into Ops and monitoring where we have our brand new LoadNinja tool for SaaS and web application load testing, as well as AlertSite for monitoring all of your applications and web assets after they've been deployed — because nothing is quite as satisfying as finding a defect before your customers do and fixing it before anyone knows that there's an issue."
He called attention to two tools in the middle of the flowchart, Zephyr and Hiptest, which are both recent SmartBear acquisitions. "These tools are vital for test management, which you need to have full insight into your testing practices and how well your manual testing efforts are integrated with any or all of your test automation tools."
Test Automation
Time, cost, and quality are three widely-recognized constraints of software project management, Greg said. "There are always going to be tradeoffs. You're either going to spend more money to increase quality and save time, or you're going to increase quality and cost yourself more time." Test automation helps conquer the constraints of time, cost, and quality in this iron triangle of tradeoffs. It is a cost-effective solution to accelerate software development while ensuring quality through earlier bug identification and quick fixes.
"We don't want to go all the way to the point of automating 100 percent of the tests as they did with Windows Vista because I think we all know how that turned out, but we want to make sure that we are automating the right things in an intelligent way," Greg said.
After a real-time poll found that the majority of audience members were just getting started with automated testing, Greg emphasized the need to concentrate test automation efforts at the unit testing level. "This is really important because that's the level where you can actually test the true functionality, whether it works, yes or no. If programmers are doing this testing as they're doing development, they're going to be able to find the defects and fix them much more rapidly."
The Automated Testing Pyramid, Greg said, is a popular strategy guide that agile teams often use when in planning their test automation strategy. As shown in the illustration above, the base or largest section of the pyramid is made up of Unit Tests, which will be the case if developers in your organization are integrating code into a shared repository several times a day. This involves running unit tests, and a variety of API and User Interface (UI) tests on every check-in. "After unit tests, you move up to API assets because more and more functionality is now powered by APIs. You can really make drastic strides in full test coverage by creating healthy and robust API tests that will allow you to say, 'Yep, our application is, for the most part, going to work.'"
"The UI testing at the top is the icing on the cake that uncovers user experience hiccups, design layout flaws, and workflow issues. This robust framework, where you have pillars of unit testing underlying API and UI tests, supports your manual testing efforts. It's all designed to give the human tester as much flexibility and time as possible to do what we do best, which is the intelligent part of testing. "The Automated Testing Pyramid is just one of many things that should be part of your organization's test automation strategy. He cautioned that there are numerous challenges in getting test automation right, citing results from a recent Zephyr How the World Tests, 2018 survey.
"There are several different ideas that we can take from this, but what it boils down to is people don't have the knowledge necessary to really implement automation... Everyone wants to automate, everyone wants to save time, everyone wants to release on a much more rapid development cycle, but they just don't know how to tackle that," he said.
In spite of these problems, test automation continues to grow year over year, Greg said, citing the results of SmartBear's State of Software Testing, 2018 Industry Report.
Benefits of Test Automation and Test Management
Greg stated that one notable fact from this year's survey is that a third of the people are still using spreadsheets and Word documents for test automation instead of a proprietary tool or integration with JIRA.
Perhaps the most surprising result of the survey is that respondents using a standalone test management tool are more likely to automate their testing across a majority of testing types, including integration testing, unit testing, UI testing, API testing, and performance testing, as compared to those who use spreadsheets and Word documents. Greg added emphasis to this point by using a comic book voice: "the kind I use when I'm reading to my kids" to read the following: "Those who invested in a test management tool automated more for every type of testing than those using spreadsheets only! "
Zephyr Overview
Milan began the next part of the presentation with an overview of the two products that make up the Zephyr test management suite: Zephyr for Jira and Zephyr Enterprise. Both products provide scalable test environments, he said, via the data center inside Zephyr for Jira and also via Zephyr Enterprise's scalable SOA- and microservices-based architecture. Zephyr has an architecture that is continuously "bashed" or tested, using APIs, virtual users, physical users, and various configurations to ensure that that the application holds up from a load testing, security, and safety perspective and can perform at scale.
He said there are quite a few approaches to reporting in Zephyr, including widgets within the Zephyr dashboard as well as live charts that are clickable to allow you to drill down to view data points in more detail. In addition to a range of traceability and custom reports, he stated that it's also possible to link up automation frameworks within Zephyr Enterprise so that you can have a consolidated view of semi-automated, fully automated, and manual testing in one place.
Since most of the webinar's audience considered themselves heavy users of JIRA, Milan said Zephyr offers multi-JIRA, multi-project support: meaning you can simultaneously connect multiple instances of JIRA and multiple projects, which can be set up within both Zephyr Enterprise and Zephyr for JIRA. It's also possible to report against all of these instances and projects by leveraging widgets available in the dashboard.
Using Zephyr and TestComplete to Do Test Automation
Milan said Zephyr offers a couple of different ways to do test automation. Zephyr Enterprise is a live HTML5 environment that connects with JIRA, and it has a whole host of applications inside it. For example, there is an artificial-intelligence (AI) engine add-on that looks at all of your requirements and testing data and helps figure out whether you're going to finish your sprint on time and what steps you can take if you're running short on time like adding additional users. It also uses a probabilistic model to ascertain whether a test case is likely to fail and flags it as a predicted fail. There's also a DevOps add-on and an add-on that allows you to work with automation frameworks using a full set of REST-based APIs to design your desired automation scripts. "Since we use these APIs to develop the UI for Zephyr in the first place, you can look at network traffic and see which APIs are being used," Milan said. You can also use the Vortex application to trigger automation jobs directly from Zephyr Enterprise.
TestComplete + Zephyr
Using Vortex, it's relatively straightforward to configure one-click automation in TestComplete and Zephyr. Vortex features include control of test executions, auto-creation of test cases, the ability to track and manage automation runs, centralized location of results for easy reporting, and analytics on testing activities. "You can trigger TestComplete suites directly from Zephyr," Milan said. "You can also automatically create test cases that have been executed using a build automation solution, a CI engine, or a CI/CD framework like Jenkins, Bamboo or TeamCity. The scripts will be auto-created inside of Zephyr Enterprise for you. For instance, if you have a job or a Java class that's being executed that runs a Selenium script, Vortex will auto-create a test case and place that into the test repository. It also places the executed version of that test case into the test case execution folder."
Milan explained that Vortex is an automation platform that uses automation agents called Zephyr Bots or ZBots, which can be installed on laptops, machines, servers, or any consolidated results location on either Linux- or Windows-based machines. Once you've installed an agent on a given machine, you have the option of setting up a job that triggers different automation frameworks directly from the Zephyr server or setting up a folder watcher that listens in on a given results location.
"In this case," he said, "the ZBot is looking at a particular location where TestComplete results are pumped in from a Jenkins build. As the Jenkins build happens, it pushes the results into the folder and the ZBot checks every 60 seconds to see whether there are new results in this folder. As soon as it finds new results, it then populates the Zephyr test planning application with new details."
"If we look at the execution results updated by Vortex, we see all the passes and indeed a failure ( red circle above ) as well. The results of any automation run can be viewed in live reports ( blue circle ), but also in a dashboards area ( green circle ). You can also have a sprint-oriented view, which shows all of your manual, semi-automated and fully automated test results in one single place."
There are a number of dashboard widgets for showing automation results including one that shows the automated status of a particular release and another one that displays what percent of tests are automated with respect to every phase or by tags in a release. This information can be published by the Daily Pulse widget that offers a running view on testing activity and tracks the progress of tests created, tests executed, and defects found.
"When you observe a test case execution failure," Milan said, "you can immediately click on a button and file a defect against any failure you've observed. These defects will go straight into JIRA for you. And then you can come back into Zephyr and build a comprehensive traceability matrix for the defect, as well."
Questions and Answers
In response to a question about Vortex in the Q&A section of the presentation, Milan clarified that Vortex is an application within Zephyr Enterprise. "It's there on the bottom left-hand side ( orange circle, above ) within Zephyr Enterprise. Vortex doesn't work directly with Zephyr for JIRA. In order to bring TestComplete results over into Zephyr for JIRA, you need to leverage ZAPI (or Zephyr API)." (ZAPI is an add-on to Zephyr for JIRA that allows access to its testing data including the ability to view and upload information programmatically.) Zephyr for JIRA users can integrate with test automation tools using ZAPI capabilities, he explained. "You can also leverage the Zephyr for Jira, Jenkins or Bamboo add-ons. In this case, the Zephyr for Jira Jenkins add-on is the better one to use since it's open source, so you'll be able to configure it and tweak the data sent over."
Greg answered in the affirmative when asked whether the Zephyr/TestComplete integration would work "headless" on a device without a graphical user interface. He referred the questioner to SmartBear's TestExecute product, which runs TestComplete tests on computers where TestComplete is not installed. TestExecute is a lightweight version of TestComplete that supports all the testing functionality provided by the Desktop, Web, and Mobile modules, but uses fewer resources than TestComplete. "This is the most common use case in a CI/CD pipeline where people will have TestExecute set up on all of their test runner machines and Jenkins on their other CI/CD platforms, which will call out to TestExecute to run the tests," he said. "All of that backend information can then be pushed into JIRA (where you can create JIRA issues for failed tests.)"
Asked whether Vortex works with Selenium, Milan said Vortex has automation triggering capabilities that let you can run Selenium scripts directly from Vortex within Zephyr Enterprise. "You can also have Jenkins trigger your Selenium scripts and put the output in a folder you've set up to listen for Selenium results."
In response to another question about Zephyr and TestComplete, Greg said that on the SmartBear product roadmap going into 2019, the company was committed to integrating Zephyr with SmartBear's TestComplete and SoapUI automation tools. "We're really taking a very thoughtful approach to how we want to integrate all of our tools into Zephyr to make sure that it's the best for all of our users," he said.
Asked about TestComplete's ease-of-use, Greg recommended that the questioner download a full feature trial of TestComplete at SmartBear.com or TestComplete.com and take it for a spin. "It's designed to be easy to use: you just have to click, record and navigate through your application. Once you're done with that, you click play and it will run the script back so you'll know how well your app runs on different browsers and different machines." SmartBear also has a plethora of video tutorials, trainings, and documentation available freely online without any login necessary.
Milan responded positively to a question about using Vortex to run Selenium scripts against code developed with AngularJS, a JavaScript-based open-source front-end web application framework. "Absolutely. It doesn't matter which application or type of application you're testing using Selenium. You can use it to test mobile platforms, AngularJS Javascript applications, or HTML5 applications. It really doesn't matter."
He concluded: "If your question is whether X, Y, or Z tool can be integrated with Vortex and Zephyr, the answer is generally 'Yes.'"
Published at DZone with permission of Tom Alexander. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments