Test Automation, API Testing, and Service Virtualization in DevOps

From Diego Lo Guidice's Continuous Delivery Conference keynote:

"For Continuous Delivery and Agile to succeed, we need to bring manual testing from 60-80% of the testing effort down to 5-20%. It's never going to disappear. Manual testing is very important—sometimes it's not worth automating something, and sometimes you just can't automate something.

How can you increase the percent of automation to about 60-80%?

API Testing

To increase automation, you have to go beyond the user interface and use API testing. UI testing is not enough for testing an application and automating the tests for distributed applications with web services, RESTful APIs, and so on. It won't help you catch everything that’s going on behind the scenes of your web or mobile app. Modern applications require a layered and decoupled architecture-driven testing approach. In fact web, mobile, connected devices (IoT), cloud, and third-party services demand a multi-tier, architecture-driven testing approach.

Moreover, Agile's fast feedback loops frequently break UI-only test suites. Agile development encourages change. When developing in short iterations, the automated UI-based test suites often break as soon as change happens, even for simple changes to the GUI layout or unexpected changes to the behavior of back-end services. Not all tools suffer from this, but most of the traditional ones do.

Download Forrester's research on API testing and functional test automation

Shift Left

To "shift left" and build quality in, you'll want to:

• Bring in your QA, testers from the very beginning of each sprint and continue throughout the process
• Incent developers to perform unit testing on all newly-developed (or modified) code
• Have testers work with product owners and business analysts from the very beginning of each sprint and continue throughout the process
  

Shift Right

"Shift right" is another term we're starting to hear more and more. Part of this is testing in production. Shift right also involves bringing in analytics and data from production and putting monitoring tools in the hands of developers so they can see how the application is behaving in production and use that to prevent problems from occurring. There's all sorts of great things happening in this space, and I think we'll be leveraging them more and more. Another thing we'll see more and more of is monitoring social networks—seeing what people are saying about the application, and feeding that back into the delivery cycle.

Simulate With Service Virtualization

With today's complex environments, how do you test continuously? You've got all these dependencies. ING has 150 scrum teams. PayPal has 680 scrum teams across the world. Just imagine all the dependencies there. There's no manager in this room that can manage that without getting hung up on dependencies.

To get around this, you need to start simulating things—for example, if you don't have access to a mainframe you need for testing, or if you're testing against a third-party service. Service virtualization can allow you to test and integrate on a continuous basis. That's how you start doing integration testing from the beginning. It's introducing simulation into the development process. What an idea! This is used in many other industries, but we haven't really used it in the software industry. I like to think about this as a wind tunnel: you put the airplane in it and test the airplane by simulating all the conditions around it. That's what we need to do—and service virtualization helps you get there."

More From the Continuous Delivery Conference

Want to learn more from Diego Lo Guidice's Continuous Delivery Conference keynote? Check out these earlier blogs and videos:

• Forrester: Why Continuous Testing is Vital for Quality @ Speed:  How testing needs to evolve in order to enable quality at speed for Agile and DevOps.
• Forrester: Barriers and Best Practices for Scaling Agile: The top 3 reasons Agile fails, how "Agile Expert" firms overcome common barriers, and Agile best practices.
• Forrester: What’s the Role of Testing in Agile and DevOps?: The importance of integrating testers and testing practices into your development teams.