Complex Functional Testing, Simplified

How does functional testing with visual assertions help simplify test development for complex real-world apps? Like, say, a retail app with inventory, product details, rotating displays, and shopping carts?

My special blog series discusses Modern Functional Testing with Visual AI, Raja Rao’s course on Test Automation University. I arrived at Chapter 6 – E-Commerce Real World Example. In this review, I hope to give you an overview of Raja’s examples and how they might apply to your test challenges.

Real-World Challenges

Raja starts by explaining the challenges in creating sophisticated functional tests for an e-commerce app. His demonstration site lets a shopper:

• Scroll inventory
• Select items
• Put items in a shopping cart
• Delete items from a cart
• Process transactions

The app includes featured items that can change each time a shopper visits the site.

In this case, Raja uses an e-commerce app from SAP, with sample data and a sample setup. The SAP application includes all the features of any e-commerce site, and it does have a simple look and feel for all that complex functionality. It's useful for demonstrating the challenges of implementing a complicated test for the app pages and functionality, which I'll dive into in a moment. But first...

You may also enjoy: 11 Best Practices to Do Functional Testing on the Cloud

There may be those among you readers who think,"Hey, this isn't a legitimate real-world test. He's using a canned application. How does that compare with complex real-world apps we code and test ourselves?"

I have to admit that I was thinking that for a second, too. And then I remembered all the times that people hesitate in applying app upgrades — even when there's a security issue. Why? Because nobody feels comfortable about how that app will behave in the areas outside the issue that got "fixed." How do we know how an app we have customized will look and behave after the upgrade? How many of us have felt burned once we discovered that the vendor's testing likely didn't include our specific configuration?

Hold that thought — I'll get back to it in a sec.

Challenges Testing Complex Applications

In the first few chapters, Raja focused on individual technologies that result in complex functional tests. He covered testing tables, data-driven testing, testing dynamic content, and testing iFrames. With each of these technologies, a small change on a single input can have a significant change on the output. For example, how do you ensure that the sort function on a table behaves as expected?

All of these tests share a common test strategy: perform an action, and then assert that the output matches expectations. Legacy functional tests require the test coder to assert each expected element exists in the DOM. Testing gets complicated by many conditions — but let's boil them down to three:

• A small change in an input makes a big change on the output that needs to be checked
• An app gets updated, and locators and formatting change
• Lots of locators exist on the app page

And, in each of the prior chapters, Raja shows how visual assertions with Visual AI result in functional testing simplified.

Visual vs. Coded Assertions

The common issues involve all the coded assertions of outputs. At a root level, do you inspect the entire DOM? Or, do you just inspect the elements you expect to change?

Some testers get functional test myopia: they focus only on the elements and behavior they expect to change in their tests. These testers think that checking every element on every page after every change seems silly. They make a change and look for expected behavior.

When you test a table sort or some other activity that changes many elements on a page, you have your hands full just writing the assertions for your expected differences. Everything else should just take care of itself.

Raja's point in each of these early chapters shows that coded assertions for DOM inspection miss all sorts of behaviors that can vary from browser to browser or app version to app version. Visual assertions with visual AI allow users to simplify their functional test code. He shows why, compared with coded DOM checks, visual assertions provide:

• Simpler deployment
• Simpler maintenance
• More robust test infrastructure

So, in chapter 6, Raja provides this analysis to testing an e-commerce app.

Testing E-Commerce App Elements

Raja begins by pointing out that many of the elements in an e-commerce app behave like elements in his earlier chapters. We find:

• A table of catalog elements with regularly placed structural parts, such as product name, description, price, and availability.
• Parts of the app screen that depend on previous activity (e.g. recently viewed products)
• App behaviors best tested as data-driven (add to cart on an in-stock vs. out of stock item)

With this example, each of the prior chapters in Raja's course comes into play.

For instance, with the catalog, the legacy approach would require the tester to identify each web element locator in the catalog section, and then ensure that the value in the catalog matched the value in the test code. With the modern approach — take a snapshot. This aligns with chapter 2 of Raja's course.

With the items in the catalog, a shopper can inspect the details of an element and then click to add it to his or her cart. The shopper can inspect both available and unavailable items — but unavailable items generate an error when the button is clicked to add the item to the cart. Testing this behavior reminds me of chapter 3 in Raja's course, the chapter about data-driven testing.

Legacy Testing — Assert Element Locators

Raja then walks through the examples and shows the legacy test code. Here is the legacy test code for the catalog test. Note that each output value must be validated.

// ------------------------------------------------------------------
// Legacy code based testing
// ------------------------------------------------------------------
@Test
public void productListTest() {
    
    // ----- Check the 1st Item -----
    By firstItem = By.id("__item0-container-cart---category--productList-0-titleText-inner");
    waitForExistence(firstItem);
    
    // ----- Assert the 1st Item's label in the list -----
    assertEquals("10\ Portable DVD Player", driver.findElement(firstItem).getText());
    
    // ----- Assert the 1st Item's Price -----
    assertEquals("449.99", driver.findElement(By.xpath(
        "//div[@id='__item0-container-cart---category--productList-0-ObjectNumber']//span[@class='sapMObjectNumberText']"))
                 .getText());
    
    // ----- Assert the 1st Item's Availability -----
    assertEquals("Available", 
        driver.findElement(By.xpath("//span[@id='status0-container-cart---category--productList-0-text']"))
                 .getText());
    
    // ----- Assert if Availability is green in color (i.e. sapMObjStatusSuccess class) -----
    assertThat( 
        driver.findElement(By.id("__status0-container-cart---category-productList-0")).getAttribute("class"),
        containsString("sapMObjStatusSuccess"));
    
    // ----- Assert the 1st Item's currency (Euro) -----
    assertEquals("EUR", driver.findElement(By.xpath(
         "//div[@id='__item0-container-cart---category--productList-o-ObjectNumber']//span[@class='sapMObjectNumberUnit']"))
         .getText());
                                           
    // ----- Assert the 1st Item has the correct image -----
    assertThat(
        driver.findElement(By.id("__item0-container-cart---category--productList-o-text")).getAttribute("src"),
        containsString("HT-2001.jpg"));
  
    // ----- Assert the 1st Item's has "Compare" link
    assertEquals("Compare", 
         driver.findElement(By.id("__attribute1-container-cart---category--productList-0-text")).getText());

    // ----- Assert if the txt Titanium exists
    assertEquals("Titanium", driver.findElement(By.id("__text20")).getText());
    
    // ----- Check the 2nd Item -----
    // TODO Repeat the above for the rest of the items
    
}

Basically, for each item in the catalog, check its:

• Label
• Price
• Availability
• Color
• Image

Compare for each against its expected reply. The code calls out to each web element locator and asserts its value.

Read this code, and then think about your own tests. Consider the similarities and differences. How much of this code remains valid between app upgrades? How much must you rewrite when development adds a new function — or a new feature? Or, if the locators change?

Simplifying Functional Testing

If you're with me so far, you understand Raja's next point — why not leave the assertions to Visual AI? In this case, the assertions let you focus on code that exercises the application and let Visual AI perform the output comparisons.

For testing the tables of inventory, Visual AI lets you perform tasks like sorting and filtering and check the output. For functions like testing what happens when you encounter success versus error conditions — like trying to add an in-stock versus out-of-stock item to the shopping cart, this is the data-driven testing scenario. As he showed in the prior chapters, and again here, Visual AI simplifies the entire test development process.

When you add new behaviors, they become new checkpoints that you can choose to include in the future baseline. And, if you encounter unexpected behaviors, you can reject those changes and send them back to your development team for repair.

As we have discussed elsewhere, visual AI provides the accuracy of AI as used in self-driving cars and other computer vision technology. Visual AI breaks a sea of pixels into visual elements without relying on the DOM to identify those elements. Once you have established a baseline, every subsequent snapshot of that screen will compare the visual elements on that page versus your checkpoint.

Instead of relying on a simplistic pixel comparison to determine whether the checkpoint and baseline differ, visual AI checks the boundaries of the element and determines whether the element itself differs in color, font, or — in the case of a photograph or other image file — image completeness.

Rather than depend on DOM differences alone, visual AI compares the rendered page with the previously rendered page.

Visual AI and Packaged Application Upgrades

As I asked earlier, how does testing a packaged application compare with testing a custom app?

As a test engineer, using the legacy approach leaves you at the mercy of the vendor. You can write all the functional tests you want, but the developer can change element locators between releases and leave you with a huge coding task — rewrite all your element locators to initiate behaviors and all your locators to measure responses.

You likely know the situation I'm describing. I know many companies that use a vendor's web service for CRM, marketing, or commerce. Once you customize one of these sites for your needs, you worry that a vendor upgrade can break your customized app. The vendor may have tested the generic app, but not the version with your custom CSS, data, and layout.

In my experience, test engineers loathe having to test packaged applications. Too often, test engineers must react to changes made in an app that the company doesn't own with tools that make it difficult to expose unexpected changes. Imagine having to use legacy tests to validate that the browser rendering remains unchanged after an upgrade. And, if the browser rendering has changed, to highlight that change and pass the information back to the app owners.

Visual AI bridges the gap for a packaged app and third-party app owners. If you have made no change and upgrade your app version, the rendering check with Visual AI makes it easy to test whether the pages have changed. If you make a CSS change or other local or global change, Visual AI ensures that the changes match your expectations.

What About Dynamic Content?

One of the conditions Raja discusses is the case of dynamic content on a page. You find this content on media sites — with layouts showing the latest stories that update regularly. You find this content on e-commerce sites as well — showing wares of interest to prospective buyers.

On the app for this chapter, it's the content that shows featured items. In the two screenshots that follow, the "Deal of the Day" rotates to multiple images, and "Promoted Items" updates each time the page gets refreshed.

This behavior makes sense — a seller wants to show a range of wares to the shopper.

Both the legacy approach and the modern approach can't handle dynamic content through automation.

Raja describes how Visual AI allows you to highlight this region as an ignore region when comparing the pages for test automation. My takeaway here is that dynamic content requires a different kind of testing - earlier in the test development process. For the functional behavior test, Applitools Visual AI lets you focus on what matters.

Conclusion: Visual AI Simplifies Functional Test

In walking through Chapter 6, you see how the legacy approach leans heavily on web locators that can change from version to version. You spend lots of resources on maintaining your tests — especially when locators change between app versions. The cost of test maintenance inclines most organizations to shy away from app version upgrades and even app behavior changes. Every time you make a change, you incur unexpected costs.

In contrast, Visual AI simplifies your functional test, while adding visual coverage. Visual AI compares rendered output against rendered output. If visual differences exist, you can identify those as intended or not. Intended changes become the new baseline. Unintended changes can go back to the app owners. And, all this is easy to manage and maintain — because you, as the tester, don't worry about the individual identifiers you need to check along the way.

Further Testing

Handling Static and Dynamic Test Objects With Katalon Studio