Generations of Test Automation Frameworks — Past and Future

In the last publication from the Design & Architecture Series, we talked about test automation frameworks and discussed all related terms. Here, we will continue the topic with the five generations of test automation frameworks. There are tons of articles discussing the different types of frameworks. I am not going to give detailed examples. I am going to try to provide you with my point of view on why they appeared in the context of the technologies of their time. Also, I am going to connect their evolution with the evolution of the QA engineer profession. At the end of the article, you will know the history of the different types of frameworks and for what you need to look into the future to be up to date.

Generations of Test Automation Frameworks

Before you begin to automate your project, you need to know the requirements; what resources you have, what you need to automate, the supported platforms, etc. However, there are different types of frameworks and you need to carefully pick one type of them. There many questions you need to answer to complete the process successfully. Here are some of them:

• How technical are the QA engineers involved?
• Who will be responsible for maintaining and developing the framework?
• Do you have someone in your team with lots of experience in designing frameworks?
• Do you have support from the DEV team?
• How complex are the functionalities you need to automate?
• Are the requirements of the applications stable? If no, how often will they be changed?

These are only a few of the questions you need to ask. I will create a separate article about them, discussing everything in more detail. My point here is that besides these questions, knowing the history and the context of the different types of test automation frameworks will help you to choose. Especially for new projects, you don't want to live in the past. There are almost no companies developing software with Waterfall and not using Agile methodologies or embracing continuous integration and continuous deployment. Also, as we are going to talk about the new generations of automation, QAs are/should be really good programmers. Sorry for writing "good" — test automation you should be technical, and whoever tells you something else is lying or trying to sell you something that costs lots of money.

1st Generation: Record and Playback

I cannot even call this type of automated test a framework since, for me, a framework should be something almost entirely coded. Some people call this type of automation a Linear Scripting Framework.

Testers don’t need to write code to create functions and the steps are written in a sequential order. In this process, the tester records each step such as navigation, user input, or checkpoints, and then plays the script back automatically to conduct the test.

The most famous example that I can give is Selenium IDE.

Pros and Cons

You don't need to write custom code. You can really fast automate some simple scenarios.

However, maintaining a large number of these tests is almost impossible. It is a big challenge to automate more complex scenarios. It is a headache to integrate these tests into a CI process, get proper reporting, and configure it for different working environments.

Context — QA and Technologies

When these tools appeared for the first time, most companies were just starting their transition from Waterfall to Agile. Back then, there were only a few QA engineers, if this was even the correct name, so most of the testing was manual since the technologies were simpler. The whole web boom was just starting, so, not surprisingly, most of these tools support web record and playback. Later, similar tools for Desktop automation appeared, such as Test Studio. However, since the releases of the new versions didn't happen so often, it was cheaper to hire more non-technical people and test the apps manually. This is why the coded solution for these folks was not going to work, since most of them didn't know how to program. With the later generations, you will see that this started to change and they had to learn new skills to keep their jobs.

2nd Generation: Modular and Data-Driven Frameworks

Modular-Based Testing Frameworks

In record and playback scripts, the data was hard-coded and more complex scenarios were almost impossible to write. This is why most vendors and open-source tools started to support an export to code option, where your recorded test is exported to a popular programming language where you can edit and modify it.

Again, the most famous example is Selenium IDE, where you could export your test to Selenium Backend code and later to WebDriver.

The whole idea behind a module framework is that the application is divided into different modules or exported scripts that you later combine into bigger tests. For example, you will have one module for login to the application, another for filling billing information, one more for submitting a ticket, and so on. In most cases, the data in the tests is again hard-coded.

Data Driven Testing Frameworks

Since hard-coding the data in the tests was limiting the combination of different steps and making the creation of new tests harder and harder, the data-driven frameworks appeared. They are just a layer on top of other types of frameworks, such as module- or library-based.

Data-Driven Testing frameworks help the user segregate the test script logic and the test data from each other. It lets the user store the test data in an external database. The external databases can be XML files, Excel files, text files, CSV files, ODBC repositories, etc.

Pros and Cons

Since these frameworks are coded, it makes the maintenance a lit bit easier, since you can use the power of programming IDEs to fix the tests. Also, they significantly increase the reuse of logic.

However, since most of the code is auto-generated, you have to rewrite it. The maintainability of the code is still low since the abstraction level is minimal, which leads to lots of copy-pasting. With the introduction of data-driven frameworks, debugging and troubleshooting got harder. Also, since the test logic and test data are separate, the tests' readability is worse.

Context — QA and Technologies

With the adoption of Agile, projects started to get bigger and more complex. People began to feel that test automation was something that could help them keep pace. But, as we discussed, because the applications got more complex, a better way of writing tests was needed.

QA folks started to learn the programming languages of their programmers and began to export tests with the hope to automate something with minimal editing. The introduction of data-driven tests helped to automate things even faster. But, the same as completing a new project, in the beginning, everyone tried to finish it no matter what, which means that nobody knew how hard it would be to keep up with this pace and maintain the tests. With time, the resources for maintaining the tests got bigger and bigger, and it took people more time to support the old scripts than write new ones.

3rd Generation: Library and Keyword-Driven Frameworks

Library Architecture Testing Framework

The Library Architecture Testing Framework is fundamentally and foundationally built on Module-Based Testing Frameworks, with some additional advantages. Instead of dividing the application under test into test scripts, we segregate the application into functions, or rather, common functions. The basic fundamental behind the framework is to determine the common steps, group them into functions under a library, and call those functions in the test scripts whenever required.

For example, during my time as a QA Architect at Telerik, we had three big websites with three shopping carts. As you can imagine, the fastest way for the developers to implement the two additional shopping carts was to copy and paste the code and change only the layout. Basically, the whole workflow stayed the same; only the front-end looked different and the web elements had different locators. Back then, we had something like a module-based testing framework. We decided that it would take us a lot of time to maintain the same tests for twp additional websites, so we upgraded our module-based test framework to a library framework, abstracting the whole workflow using abstract facades. Page objects are a significant part of this type of framework.

Keyword-Driven Testing Frameworks

In a keyword-driven framework, each function of the application under test is laid out in a table with a series of instructions in consecutive order for each test that needs to be run. In a similar fashion to data-driven frameworks, the test data and script logic are separated in a keyword-driven framework, but this approach takes it a step further.

In the table, keywords are stored in a step-by-step fashion with an associated object or the part of the UI that the action is being performed on. For this approach to work properly, a shared object repository is needed to map the objects to their associated actions.

Pros and Cons

Though the library based frameworks much more test code can be reused. The fixes start to happen in a single place.

However, with time if there are slight modifications to the workflow in some of the apps, it gets harder and harder to keep the abstraction good. Writing more abstract code requires engineers to have more in-depth programming knowledge. Because of the higher abstraction, the tests get even harder to read.

Keyword-driven frameworks had to ease the test writing process, however, they made the maintenance even harder and slowed down the development of new tests since the people who write the tests are not the ones responsible for maintaining the keyword repository.

Context — QA and Technologies

The library-based testing frameworks were the natural continuation of the module-based frameworks. Since applications started to get even more complicated and release cycles were shortened significantly, it became clear that to keep up with the new functionality development and keep maintaining the old tests, something had to be done. These frameworks provided a small step forward for faster test creation and better maintainability.

On the other hand, there were still many companies that hadn't embraced test automation and continued to hire non-technical QA people. However, at some point, they realized that to cope in the new world, they had to start embracing the Agile and creating automated tests. However, until their manual QA engineers learn the new required skills something had to be done, this is how the keyword-driven testing frameworks appeared. Developers got involved to help, building the "core" keyword repository and providing the QA engineers with a way to write tests in a non-technical way. This way was okay ten years ago, but I don't know a single company that continues to do this and has healthy test automation.

4th Generation: BDD and Hybrid Frameworks

BDD Testing Frameworks

Behavior-Driven Development frameworks allow automation of functional validations in an easily readable and understandable format for business analysts, developers, testers, etc. (or at least, this is the marketing message). Such frameworks do not necessarily require the user to be acquainted with the programming language. There are different tools available for BDD like Cucumber, SpecFlow, Gauge, etc.

Hybrid Testing Frameworks

As the name suggests, a Hybrid Testing framework is a combination of more than one of the above frameworks. The best thing about such a setup is that it leverages the benefits of all kinds of associated frameworks.

Pros and Cons

A hybrid framework can be more easily adapted to get the best test results. However, hybrid solutions still have the same drawbacks as all other frameworks that they are built on top of: not-so-readable tests, not-so-easy to maintain and troubleshoot, and usually built to support only a single platform or technology.

BDD frameworks have the same pros and cons as the keyword-driven frameworks. The only difference between them is that the BDD markups are integrated more tightly with the programming IDEs.

Context — QA and Technologies

Hybrid frameworks, or so-called "custom" testing frameworks, started to appear in almost every company. This is maybe the most popular approach nowadays, which is entirely understandable. As pointed out before, most projects are altogether different, so there couldn't be a general tool or framework that can automate them all without any kind of modification. This is why hybrid testing frameworks are so popular.

BDD appeared as a continuation of data and keyword driven frameworks. There are still companies with a significant amount of non-technical people. It is believed that business people will write tests, but in practice, nobody does that. People just continue to ignore the fact that this is an unnecessary abstraction that drastically slows down development time and makes maintenance of the tests much harder. You can check out this conference talk about the subject (articles) where we use the scientific method to measure the benefits of different frameworks.

4.5th Generation: Selenium-Based Frameworks

Selenium-Based Testing Frameworks

Until now, all testing framework types were more or less focused on how we write the tests. Some people decided that to make the framework "core," it would be good to share it with the rest of the world and open-source it. You can find plenty of examples, like Selenide, Atata Framework, EPAM JDI, and Objectivity. You can base a library or hybrid framework on them and get core benefits such as more readable tests, less brittle tests (usually they include built-in logic for waiting elements), and slightly better test maintainability. The primary focus of these frameworks is web automation. You are free to add any abstraction you want, like keywords or BDD.

Pros and Cons

I believe that the open-source Selenium-based testing frameworks are great. Top recognized automation QA engineers build them. So, for sure, it is much better to base your hybrid framework on them than reinvent the wheel.

However, they have drawbacks, too; they are built primarily for web, so if you need a solution for API, mobile, or desktop, you have to create it yourself. All of these frameworks lack proper documentation. If you have a problem, in some cases, the code is complicated, so you need to submit an issue. But since these guys are developing the tools in their free time, you don't know when they will respond. Even if they do, you cannot be sure whether the issue will be fixed or when. Almost all of these frameworks only wrap WebDriver, making the tests more stable without giving many new features (which is better than nothing, of course). This is the case since these guys are doing it mostly in their free time; they don't have the time to make significant developments or spend much time thinking how to make the API more useful or write unit tests.

Context — QA and Technologies

Since website automation has been here for the longest, the open-source "core" Selenium-based testing frameworks appeared. If you check their websites or GitHub pages, you will see that outsourcing companies build them. Their goal is, of course, winning more customers by promising that the projects they create will be well tested and that at the end of the project, they will get the framework and all tests. Another part of these frameworks is custom frameworks built for a private company that some of their QA engineers want to popularize and show how good it is. There are benefits for the company as well since some new features are added to the framework without paying to someone to do it. The third type is where someone in his free time decided to build an open-source test framework, but he continues to work for another company. This is like a hobby project. These folks earn some cash by providing support to some of the companies that decide to use their free tool. Everyone loves free, but sometimes the cheap stuff is much more expensive (we will talk about that in another article).

5th Generation: Rise of AI, Full-Stack Frameworks

AI Tools — "Fake" Future?

Nowadays, "AI" is a hype word. Everyone that wants to sell you something includes it in front of their tool or in their marketing materials. There are promises for systems that can write automated tests automatically, codeless tests, or effortless/almost zero test maintenance. I cannot say that all of it is bullshit — there are some decent tools such as AppliTools (they build tools for visual testing). These tools will have their place in the future, but they can only solve some edge case scenarios and won't be able to replace the coded tests.

Full-Stack Test Automation Frameworks: True Future/Present

So, considering the context, what would be the best 5th-generation testing framework?

• It should not be a hobby project or side project of some outsourcing company. It should be professionally built and tested. For me, a test automation product should be treated as a product. Otherwise, how you can trust it and rely on it?

• Since building a framework is something interesting everyone even people with less experience want to build one. I get it. Nobody wants to have a boring job. But don't worry no matter which generic framework or tool you use, you still need to integrate it. However, I firmly believe that the toughest part of the test automation framework should be built by an architect that read tons of books about designing APIs, high-quality code and automated tons of applications.

• The new generation frameworks should be well documented.

• The new generation frameworks should be fully tested with all kinds of tests. (which for me questions the whole open source thing)

• The new generation frameworks should be fully coded- no more keywords, sentences, etc. QA folks should embrace their destiny that they will have to learn to code.

• It should be designed to support different platforms- web, mobile, desktop and API, all kinds of resolutions, browser versions, etc.

• It should be cross-platform- you don't have to limit the user which platform they use- Mac, Linux or Windows.

• Parallel and distributed testing are hard to implement so this generic framework should give this by default to people.

• This is no more art of amateur programmers, test automation frameworks should be treated as production code, so this 5th generation framework should enforce high-quality coding standards.

• One of the hardest things to develop is to allow these generic frameworks to be extendable and customizable. All of the mentioned ones lack this property which is crucial if you are going to integrate a framework written by somebody else. Every company and team should be able to modify it and extend it to fit their needs.

• With the increasing tests count and complexity, it will be even more critical the tests be more maintainable. A significant part of this effort is the easier troubleshooting and better support for locating errors.

• Documentation is not enough. I bet that most of you that have more than few years of experience in the field of automated testing and have a custom framework spent countless hours teaching new colleagues how to write "proper" tests using the team's framework. I believe that there should be a more automated process allowing people to learn by themselves. I think one way to do it is to utilize the so-called getting started solutions or starter kits. Projects that have examples with explanations on how to write tests and why we use a particular method or not.

Do such frameworks exist? Yes. Some tool vendors already provide complete systems, such as SmartBear and IBM. However, their tools don't always conform to the above statements.

Being one of the best in the region led me to consult many local companies regarding their automation efforts. In the process, we discovered that most of them needed such 5th-generation test automation framework. Building such a tool is a complex task and takes lots of time — needless to say, it won't be possible to maintain different versions of the same thing for ten different companies. This is how Bellatrix Test Automation Framework was born. It took us almost two years to build, but I am pleased with the result. It is a 5th-generation framework conforming to all of the above statements. Even if you are not searching for something new, I still encourage you to try it and install the starter kit. I promise that you will get lots of new ideas on how to improve your existing framework (even only by reading the docs).

Pros and Cons

For me, AI frameworks are just the new buzzword and "BDD" beliefs that you can write automated tests without programming, trying to sell something to QA folks that still refuse to accept that they will need to learn to code. As mentioned, there are some excellent tools that solve some particular problems, but I haven't seen any of them be sufficient enough to replace the hybrid frameworks.

The full-stack frameworks have countless benefits and are much better in many ways than the previous generation of frameworks and tools. However, there is always a catch. To use the new generation frameworks, QA engineers need to be much more technical. They need to find time to learn new programming languages and read even more about high-quality code, code design, and architecture.

Even having the best generic framework that gives you all of the above benefits won't solve many of the current QA folks' automation struggles, such as having a "clean" test environment, preparing the "right" test data, designing the "right" tests, how to report the results in the best possible way (yes, I still believe that reporting is not part of the frameworks, you can integrate something but you should have a dedicated tool/system for it), and so on.

Context — QA and Technologies

With the rise of mobile development, microservices, and cross-platform technologies, the requirements for test automation frameworks got more sophisticated. Now it is not enough for your website to run okay on the latest Firefox on Windows. You have to support the latest five versions of six different browsers and you have to test the layout of your website on various mobile devices. Also, most companies nowadays have Android and iOS native apps, so your framework should be able to test them as well. In order for the application code to be reusable, companies are shifting to microservices, which are based on containers and APIs. Thus, you have to test the APIs as well. Most frameworks nowadays are based on the "standard tools" such as Selenium WebDriver, Appium for the web, and WinAppDriver for Windows desktop apps. Moreover, the period of rushing to write a huge amount of automated tests is almost at its end. Nowadays, one by one, people are starting to realize that it is more important to have fewer but more stable tests. This is why the new wave of test automation frameworks will have to answer to this need, providing better tools for troubleshooting, maintainability, readability, and coding standards.

Summary

As you saw, there are many different varieties of frameworks. Because of their long past, many companies continue to support some legacy types of frameworks because it is too hard to rewrite them. Is this right or wrong? I cannot say. I know that time — and people — are money. It is hard to fire people because they don't have the right knowledge to support the future systems. As you read, since almost the beginning, there have always been two high-level options: one with code and one without code. The second is for the companies that I mentioned here at the beginning of the paragraph, full of people that still need to re-educate themselves. I cannot say that this is something wrong — it is as it is. This is just an observation.

If you have read more than one of my articles, you know that I firmly stay behind the idea of coded tests, and a big part of why I write these series of publications is to show you the future from my perspective. In my opinion, all people who want to do "proper" test automation have to start to educate themselves, learn to write high-quality code, stop being "amateur" programmers, and embrace the future.