Introducing PICQT for Writing Cucumber Tests With Iridium: Part I

If you have taken the time to write an Iridium test we hope you have found that it makes it easy to document and validate a simple journey through your web applications. The Cucumber steps included with Iridium have been designed to make clicking, typing and navigating around a web application as straightforward as possible, and using the snippets package for Atom, you’ll have your first end to end test written up in no time.

However, you will reach a point where your Iridium tests need some structure in order to fulfill the sometimes competing demands of a modern development environment. After having written tests for a number of web applications, we have found that they typically need to support the following use cases:

• Run detailed regression tests covering a range of edge cases.
• Run quick tests to highlight obvious problems with applications or infrastructure.
• Automatically click through an application to get to the point where developers are actively working on an application.

To accommodate these use cases, we have developed the PICQT (pronounced “picked”) system. PICQT is a standard set of tags that are applied to scenarios in an Iridium test script in order to support the use cases listed above.

PICQT stands for:

• Profile - tags for scenarios that configure Iridium performance options
• Individual - tags for scenarios that test individual components of a web application, and tag sets that are used by Iridium to target individual scenarios
• Complete - tag sets that instruct Iridium to run a complete test
• Quick - tag sets that instruct Iridium to run a quick test
• Terminal - tags for scenarios that leave the application in a state where no additional testing can be performed

Before we start, download the latest version of Iridium here.

To see how these tags work in practise, let’s take a look at a test script that implements the PICQT standard.

Feature: Test Home 2.0 Quote (HQS2.0)

    @pageObject
    Scenario: Generate Page Object
        Given the alias mappings
            | AddressInput               | [id=address-typeahead]                   |
            | AddressSelector            | [title='16 Bent Street, City Beach, WA'] |
            | commencementDate           | [name=commencementDate]                  |
            | ConstructionDate1914Button | .//button[@value='1914-01-01']           |
            | ContentsSumInsured         | [type=tel]                               |
            | ContinueButton             | [data-role=button-next]                  |
            | HomeAndContentsCoverXpath  | .//*[text()='Home & Contents']           |
            | HomeSumInsured             | [type=tel]                               |
            | InternalSiren              | [value=securityAlarmHasInternalSiren]    |
            | No                         | [value=false]                            |
            | Yes                        | [value=true]                             |
            | OtherMetalButton           | [value=OTHER_METAL]                      |
            | OwnerOccupiedButton        | [value=OWNER_OCCUPIED]                   |
            | PostcodeInput              | [name=postcode]                          |
            | StoneButton                | [value=STONE]                            |
            | VillaButton                | [value=VILLA]                            |
            | YearMovedIn5YearsAgoXpath  | .//*[text()='More than 5 Years']         |

    Scenario: Default Profile
        These are the default settings used for a test run. The wait times are
        generous to ensure the test will pass.

        When I set the default wait time between steps to "2"
        And I set the default wait for elements to be available to "10" seconds
        And I set the default wait for elements to be available to "30" seconds
        And I set the default keystroke delay to "0" milliseconds
        And I save the current date with the format "dd-MM-yyyy" to the alias "Today Date"

    @profile @quick-profile
    Scenario: Quick Profile
        These are settings that developers can use to quickly run a test in
        development.

        When I set the default wait time between steps to "0.5"

    @home-launcher
    Scenario: start HQS application
        And I delete all cookies
        And I open the application
        And I wait "30" seconds for the element found by "ContinueButton" to be displayed

    @detailed-test @capture-address-detailed-test
    Scenario Outline: Capture postcode and postcode validation
        And I clear the element found by "PostcodeInput"
        And I populate the element found by "PostcodeInput" with "<postcode>"
        Then I verify that the page contains the text "Invalid postcode"
        Examples:
          | postcode |
          | 1000     |
          | 2699     |
          | 0799     |

    @exit-step @capture-address-detailed-test
    Scenario: Exit after capture postcode and postcode validation
      And I skip all remaining steps      

    @capture-address
    Scenario: Capture Address
        And I clear the element found by "PostcodeInput"
        And I populate the element found by "PostcodeInput" with "6015"
        And I populate the element found by "AddressInput" with "16 b"
        And I click the element found by "AddressSelector"
        And I click the element found by "ContinueButton"

    @capture-property-type
    Scenario: Capture property type and details
        And I click the element found by "VillaButton"
        And I click the element found by "No"
        And I click the element found by "ConstructionDate1914Button"
        And I click the element found by "StoneButton"
        And I click the element found by "OtherMetalButton"
        And I click the element found by "OwnerOccupiedButton"
        And I click the element found by "YearMovedIn5YearsAgoXpath"

    @commenceDate
    Scenario: Capture policy type and commence data
        And I click the element found by "HomeAndContentsCoverXpath"
        And I select alias "Today Date" from the drop down list found by "commencementDate" if it exists
        And I click the element found by "ContinueButton"

    @detailed-test @detailed-test-rebuild-cost
    Scenario Outline: Home rebuild cost validation
        And I clear the element found by "HomeSumInsured"
        And I populate the element found by "HomeSumInsured" with "<amount>"
        And I click the element found by "ContinueButton"
        Then I verify that the page contains the text "<message>"
        Examples:
          |amount  |message                                             |
          |1000001 |Your rebuild cost cannot be higher than $1,000,000  |
          |149999  |Your rebuild cost must be at least $150,000         |

    @exit-step @detailed-test-rebuild-cost
    Scenario: Exit after test home rebuild cost validation
      And I skip all remaining steps

    @rebuild-cost
    Scenario: Capture rebuild cost
        And I clear the element found by "HomeSumInsured"
        And I populate the element found by "HomeSumInsured" with "1000000"
        And I click the element found by "ContinueButton"

    @heritage-list
    Scenario: Capture heritage listed status
        And I click the element found by "No"

    @security-alarm
    Scenario: Capture security alarm System
        And I click the element found by "Yes"
        And I click the element found by "InternalSiren"
        And I click the element found by "ContinueButton"

    @detailed-test @replace-cost-detailed-test
    Scenario Outline: Contents replacement cost validation
        And I clear the element found by "ContentsSumInsured"
        And I populate the element found by "ContentsSumInsured" with "<replace_amount>"
        And I click the element found by "ContinueButton"
        Then I verify that the page contains the text "<message>"
        Examples:
          |replace_amount    |message                                                 |
          |500001            |Your replacement value cannot be higher than $500,000   |
          |99999             |Your replacement value must be at least $100,000        |

    @exit-step @replace-cost-detailed-test
    Scenario: Exit after contents replacement cost validation
      And I skip all remaining steps  

    @replace-cost
    Scenario: Capture content replacement cost
        And I clear the element found by "ContentsSumInsured"
        And I populate the element found by "ContentsSumInsured" with "500000"
        And I click the element found by "ContinueButton"

The “I” in PICQT

Let’s start with the “I” in PICQT, which gives developers an easy way to quickly jump to the point in the application that they are actively developing by focusing on an individual scenario. In this example, that would be the part of the application where we are asking for the cost to rebuild the house.

As you can see, there are half a dozen screens to populate before the “rebuild cost” screen is displayed. Anyone who has ever worked on a web application will have endured the experience of making a minor change and then clicking their way back through the application to the page they are working on. It is tedious and time-consuming, so why not automate it?

There are 3 tags that we need to add to allow Iridium to prepopulate the application and then return control to the developer.

@detailed-test

The @detailed-test tag is used to identify scenarios that run through a detailed set of tests. Often you’ll be verifying input validation, ensuring that navigating back and forth works as expected, or validating the functionality of a complex widget.

In our example, the scenario outline “Home rebuild cost validation” has the @detailed-test tag to indicate that it will perform a series of validations against the text box that holds the value entered by the user.

@<step-name>-detailed-test

The @detailed-test-<step-name> tag is used to identify a particular detailed test scenario. It is always paired up with the @detailed-test tag or the @exit-test tag.

In our example, the @rebuild-cost-detailed-test tag has also been assigned to the the scenario outline “Home rebuild cost validation”.

@exit-test

The @exit-test tag is assigned to a scenario that looks like this:

 @exit-step @rebuild-cost-detailed-test
 Scenario: Exit after answering home rebuild cost validation
   And I skip all remaining steps

The step And I skip all remaining steps is essential an early exit from the test script. Any steps after this one return immediately without performing any actions.

Scenarios with the @exit-test tag exist to exit out of the script and give control of the browser back to the developer. This is how we can utilize an end to end test to give developers quick entry points into the application as they are developing it.

Running the Test

To run the test, stop it at the “rebuild cost” screen and return control to the developer, run the following command:

java \
    -DmonochromeOutput=false \
    -DtagsOverride="~@detailed-test,@rebuild-cost-detailed-test;~@exit-step,@rebuild-cost-detailed-test;~@profile" \
    -DtestDestination=Chrome \
    -DenableScenarioScreenshots=true \
    -DtestSource=./test.feature \
    -DsaveReportsInHomeDir=true \
    -DleaveWindowsOpen=true \
    -DstartInternalProxy=-browserMob \
    -DappURLOverride=https://application.com \
    -jar IridiumApplicationTesting.jar

There are three important settings in this command.

The first is the list of tags that Iridium will use when running the test. The string ~@detailed-test,@rebuild-cost-detailed-test;~@exit-step,@rebuild-cost-detailed-test;~@profile looks imposing, but is actually quite striaght forward.

First, the string is broken by semicolons into tag groups that and ANDed together. So this string means (~@detailed-test,@rebuild-cost-detailed-test) AND (~@exit-step,@rebuild-cost-detailed-test) AND (~@profile).

Next, the string is broken down by comma and ORed together. So the next step means (~@detailed-test OR @rebuild-cost-detailed-test) AND (~@exit-step OR @rebuild-cost-detailed-test) AND (~@profile).

The tilde means that a tag must be absent on a scenario for the condition to be true. So in plain English this means  (scenario must not have a @detailed-test tag OR must have a @rebuild-cost-detailed-test tag) AND (scenario must not have a @exit-step tag OR must have a @rebuild-cost-detailed-test tag) AND (scenario must not have a @profile tag).

In effect, this tag set means that we don’t run any scenarios with a @detailed-test tag or a @exit-step tag unless the scenario also has a @rebuild-cost-detailed-test tag (we’ll cover the @profile tag later).

The second important setting is the leaveWindowsOpen system property, which is set to true, meaning Iridium will leave the browser window open after the test has completed.

The third important setting is the startInternalProxy system property, which is set to -browserMob, meaning that Iridium will not create an instance of the BrowserMob proxy that is usually created by default. We disable this proxy because we don’t want to leave a browser window open that is configured to use a proxy that will be shutdown when Iridium exits. If we didn’t disable this proxy, the remaining browser window would be inoperable as it tried to direct network requests to a proxy that no longer exists.

What this all means is that the script will run the scenarios that progress the web application up to the  “rebuild cost” screen, run the detailed tests written for the “rebuild cost” screen, skip the rest of the scenarios thanks to the step in the @exit-step scenario, and leave the browser open for the developer to continue manually testing. This can speed up development dramatically as developers have a single command to run in order to quickly jump them to the page that they are working on, and has the added benefit of also ensuring that the test script is kept in sync with the latest changes to an application.

Stay tuned for Part II, coming soon!