Selenium Versus Karate: A Concrete Comparative Approach

In the world of software architecture, which is still in its infancy in absolute terms, change is still rapid and structural. Deep paradigms can be overturned, changing the structure of information systems. As coupling has become a key issue for many information systems, new architectures have emerged, notably SOA (Service Oriented Architecture), followed by microservices. These two fairly widespread architectures certainly make it possible to solve coupling problems, but as always there are certain trade-offs to be made. The complexity of testing is one of them.

Like a balanced equation, an information system is an equivalence between a technical expression and a functional expression, and if one changes, we need to be able to guarantee the validity of the equivalence between the two. To attain this, we need to be able to test all parties using tools that can establish this equality. When releasing a new version of a microservice, it’s fairly easy to run unit and even integration tests before and during deployment to validate its internal operation. 

But how do you guarantee that its integration with the other microservices will always be valid? How can you be sure that a system made up of a constellation of microservices will behave as expected if you only run tests on each component? How can we guarantee that the new technical expression of our equation will always give the same functional expression? By testing it.

It can be manual, of course, but it can also be automated*. Let’s take a look at this automation, using two technologies we’ve used, Selenium and Karate. The aim of this study is not to make a theoretical comparison, of which there are so many, but a concrete one. If a developer today wants to use behavior-driven development, what will he have to do with one of these options?

The study will first provide a quick analysis of the functionalities offered by both frameworks. We will then delve into the technical aspects, using a specific use case with a focus on Programming and CI/CD. Finally, we will examine the communities surrounding both frameworks. Selenium will not be studied on its own; in order to compare a level of functionality equivalent to that of Karate, it will be used with Cucumber. This will make it possible to test technical packages that allow automatic tests to be written in natural language, thus satisfying a BDD requirement. In our case, we will opt for the Java version of Selenium, although other alternatives do exist.

Features

Selenium/Cucumber

Selenium

• Selenium IDE: Enables recording of actions performed on a browser, this
  Firefox plug-in saves recorded scenarios as ”side” files for future use.
• Selenium WebDriver: This is a toolkit for interacting with different web browsers using the Gecko and ChromeDriver drivers. We’ll be using this toolkit if we opt for Selenium. It’s available in several languages, including Java, JavaScript, and Python.
• Selenium Grid: Enables WebDriver scripts to be executed on remote (or real) machines by sending commands from the client to remote browser instances. The aim is to provide a straightforward method of running tests concurrently on multiple machines.

Cucumber

Cucumber is an open-source tool for behaviour-based testing BDD. It describes expected software behavior in a natural language that can be understood by both technical and non-technical stakeholders. This language is referred to as Gherkinand is used to explain functionalities in a clear and structured manner. Each test can be automated through code (automated behavior using Selenium). This program is known as glue code and can be written in various languages such as Java, Csharp, and Ruby, among others. However, adhering to the specifics outlined in the introduction, we will focus solely on the Java implementation. It can also produce comprehensive execution reports to facilitate the reading of test execution outcomes.

Karate

This framework was originally based on Cucumber until its release 0.8.0, when it was separated from it. This decision proved to be beneficial. Nevertheless, it still uses Gherkin expressions for improved clarity, readability, and test organization similar to Cucumber.

• API test automation: Karate’s initial foundation is the creation of API tests from Gherkin files. Other features have been subsequently integrated to enhance its capabilities. It is a direct competitor to REST Assured.
• Mocks: This section facilitates the generation of API mocks, which are highly advantageous in microservice scenarios or for separating front-end and back-end teams.
• Performance testing: Based on API testing, Karate incorporates Gatling to avoid having to rewrite user flows, by applying performance testing to pre-existing API tests.
• UI automation: Finally, Karate provides UI tests that automate user behavior by interacting with the DOM. These tests are written in Karate DSL, based on the Gherkin language.

Programming

Use Case Description

• Open a Google browser page.
• Search for Martin Fowler.
• Click on the first result that contains Martin Fowler.
• Check that you are on ”https://martinfowler.com/”.

Selenium/Cucumber

Here, we will work in three stages.

1. Write the Gherkin Cucumber scenarios that describe the test cases.
2. Create the Glue Code to link the previous scenario steps to the code using the Cucumber framework.
3. Use the Selenium library to interact with the browser and write any necessary utility functions.

Gherkin:

Feature: Demonstration use case
Scenario: search for Martin Fowler website
Given I navigate to ’https://google.com’
And I search ’Martin Fowler’ in google
search bar
When I click on result containing
’Martin Fowler’
Then the current url is
’https://martinfowler.com/’

Glue Code:

@When("ˆI navigate to \"([ˆ\"]*)\$")
    public void navigate_to_url(final String
                                        urlToTest) {
        navigateToUrl(urlToTest);
    }

    @When("ˆI search \"([ˆ\"]*)\ in google
            search bar$")
            public void search_data_in_google(final
            String searchText) {
        WebElement element =
                getElementByName("q");
        fillElementWithText(element,
                searchText);
        clickElementByContain("Recherche
                Google")
    }

    @When("ˆI click on result containing
            \"([ˆ\"]*)\$")
    public void click_on_first_result(final
                                      String searchText) {
        clickElementByContain(searchText);
    }

    @When("ˆthe current url is \"([ˆ\"]*)\$")
    public void current_url_test(final
                                 String urlToTest) {
        checkCurrentUrl(urlToTest);
    }

Then we will utilize the Selenium library. The Selenium Toolbox includes this.config.getDriver(), allowing access to functions like navigate() or findElement(…).

public void navigateToUrl(final String url) {
        this.config.getDriver().navigate().to(url);
    }

    public WebElement getElementsByName(final
                                        String name) {
        return this.config.getDriver()
                .findElement(By.name(name));
    }

    public void checkCurrentUrl(final String
                                        urlToCheck) {
        assertEquals(this.config.getDriver()
                .getCurrentUrl(), urlT);
    }

    public void clickElementByContain(final
                                      String contain) {
        WebElement element = this.config.getDriver().findElement(
                By.xpath(String.format("//[contains(text(),'%s')]", contain);
        element.click();
    }

Karate

With Karate, tasks are much faster: all you need is a scenario file using the Karate DSL (Domain-specific language) to achieve the same desired outcome.

Feature: Demonstration use case
Scenario: search for Martin Fowler website
Given driver ’https://google.com’
And input(’input[name=q]’, Martin Fowler)
And click(’{ˆ}Recherche Google’)
When waitForText(’Martin Fowler’,’APPEARED’).click(’{ˆh3}MartinFowler’)
Then match driver.url ’https://martinfowler.com/’

Analysis

Here we can see a distinct difference in the amount of code required. Karate has already integrated the DOM interaction functions down to the Gherkin language level, which is a significant advantage in terms of development speed. However, this may affect the readability of the scenario file, particularly in a BDD context. As a result, it is reasonable to question whether BDD can be effectively executed using Karate.

The answer may vary depending on the project’s context, its users, and the technical expertise of those involved.

However, using Karate can greatly reduce maintenance expenses due to having less code and fewer bugs. This is a critical factor in the profitability of automated testing, which is contingent upon its simplicity, maintainability, and durability.

CI/CD, Performance and Scalability

In both cases, we presume that we will be using the following
process:

Basic automation process

The issue of data is not relevant in our case. Although it is an important factor when discussing test automation, both Selenium and Karate encounter the same problem and it is unrelated to their core functionality.

So, our main focus will be on how both technologies can be integrated into a CI/CD environment.

Selenium

Here we will explore the use of Selenium Grid to compare the full range of features offered by Selenium.

Required Components

• Selenium Grid Hub: The central control point of the Selenium Grid architecture which manages the distribution of test execution to different nodes (machines or virtual environments). The hub receives test requests from test scripts and routes them to available nodes according to desired capabilities, such as browser, platform, version, etc.
• Nodes: They are individual machines or virtual environments that are responsible for executing the tests. Each node registers with the hub and advertises its capabilities, including supported browsers and operating systems. Test scripts connect to the hub, which in turn redirects them to appropriate nodes for execution based on the desired capabilities.
• WebDriver Instances: WebDriver instances are indispensable for interacting with browsers and automating UI tests. The Remote WebDriver instance is used in the test script to send commands to browsers running on the nodes. These instances act as a bridge between the test script and the browser, enabling actions like clicking, inputting text, and validating content.

The architecture of the aforementioned components is as follows:

Selenium Grid components architecture

Another option is to use Selenoid, an open-source project that offers a lightweight and efficient method for implementing Selenium Grid through Docker containers. It simplifies the process of running Selenium tests across various browsers and versions. Selenoid brings containerization benefits to Selenium Grid, which facilitates the handling of test execution environments and reduces resource overhead. Selenoid also offers built-in video recordings of test sessions. This is especially helpful for diagnosing test failures, as you can watch the video to comprehend the failure context.

Selenium tests execution on Selenoid

The key distinction lies in the technology employed. Selenoid utilizes Docker containers to achieve browser isolation, whereas Selenium Grid relies on separate nodes with Remote WebDriver instances. The objective of both approaches is to furnish uniform and replicable browser environments for test execution, alleviating problems that may crop up due to shared browser instances.

In summary, both Selenium Grid and Selenoid utilize specialized browser instances for every test session to guarantee a stable and separate testing environment. Though approaches may vary, the fundamental principle of browser isolation persists.

Karate

For Karate things are much more simple. Two Docker images are available and should be deployed on the CI server in order to emulate the browser. Then you can deposit your Karate scenarios and launch them in different manners:

• Using a standalone version of Karate (In that case you
  will prefer to use this Docker image)
• Using a Java jar containing the Karate library

Karate CI/CD architecture with standalone Karate jar

It is important to note that Karate enables native multithreading. Instead of using multiple browser instances to run tests, tests can be executed concurrently by adding a custom parameter. The figure below shows a multi-threading inside a container with three threads.

Karate multi-threading

Communities and Usage

It is noteworthy that when it comes to e2e, Selenium is the leading framework and enjoys widespread adoption in the community. Therefore, we will commence by conducting a comparative analysis, followed by a closer examination of the activity surrounding Karate.

Comparative Analysis

GitHub Stars

This initial metric measures the amount of "stars" granted to various repositories by the GitHub community. However, this criterion alone is not conclusive as bots may artificially inflate the value. As a result, we utilized the Astronomer tool, which provides a confidence score for Github repositories based on the subsequent criteria:

• The average amount of lifetime contributions among stargazers
• The average amount of private contributions
• The average amount of public-created issues
• The average amount of public authored commits
• The average amount of publicly opened pull requests
• The average amount of public code reviews
• The average weighted contribution score (weighted by making older contributions more trustworthy)
• Every 5th percentile, from 5 to 95, of the weighted contribution score
• The average account age, older is more trustworthy

Analysis of the Intuit/Karate repository with Astronomer

The achieved grade of "A" confirms the quality of the information analyzed within the repository. Therefore, we consider the ”stars” criterion as reliable. The figure below displays Cypress and Cucumber for additional comparison points besides the two examined frameworks. The y-axis represents the number of Github stars, and the x-axis shows the date. Expectedly, Selenium surpasses its rival. However, it is worth mentioning that Karate has gained significant ground and even outperformed Cucumber, which is a highly prevalent framework utilized for BDD development with Gherkin. Cypress remains popular, particularly within the JavaScript community, due to its significant reputation.

Comparison of the number of GitHub Stars — Star history

StackOverflow Trends

We will now examine the "trends" criterion on Stack Overflow to gauge the activity of the community involved in a particular technology. By correlating the number of users with the corresponding tag on Stack Overflow, we can assess the level of support available for the technology, as the site is extensively used by the developer community. This ensures varying levels of support (courtesy of the community, given that these are open-source projects). The greater the frequency of occurrences, the simpler it is to discover solutions to specific problems. The initial graph examines the following technologies: Selenium, Cucumber, Cypress, and Karate. The y-axis presents the proportion of questions posted on Stack Overflow that contain the corresponding tag, while the x-axis displays the months/years.

Stack Overflow trends - including Selenium

Once again, Selenium is in the lead, confirming the previous result. To improve our analysis, we will display the same graph without Selenium to avoid compressing the curves (the drop in the Selenium curve is due to the fact that it has been moved by StackExchange to another website dedicated to software quality).

StackOverflow trends — without Selenium

Karate has a high percentage, experiencing a significant rise since its inception. Cucumber has remained stable, closely trailing Karate. Cypress is still on top but seems to know an important decrease. A correlation can be established between the acceleration depicted on the ” GitHub Stars” chart and the level of occurrence here.

Conclusion

We note that Karate is a more code-efficient framework, enabling simple writing due to its design to avoid Selenium’s complexity. Its CI/CD capabilities are powerful for most projects. However, Selenium Grid still offers specific features that Karate does not for certain integrations. The established and strong community around Selenium is a valuable aspect, as is the variety of supported programming languages. On the other hand, Karate only offers one ”language” — its own DSL. Whilst this is quite easy to learn and intuitive for programmers, it can still be a bit complicated for non-technical users, especially in a BDD context. The community around this framework is growing, and many improvements have been made since its inception. Peter Thomas is very responsive on Stack Overflow and his own GitHub, answering questions promptly, and the extensive documentation is clear and exhaustive. However, the project is still very closely linked to Peter Thomas for the moment. Also, it’s crucial to bear in mind that we’re solely referring to Karate-UI and that this framework provides several other functionalities, such as API testing and performance testing using Gatling based on these API tests, which is highly engaging. Karate is a contemporary and intriguing testing framework that presents a viable option to consider for your project, depending on its specific characteristics.