How to Stop Test Suite After N Test Failures In Pytest
At times, test scenarios are interdependent and one failure causes other failures. Avoid such cases in this pytest tutorial with maxfail and pytest.mark.incremental.
CORE
·
Join the DZone community and get the full member experience.
Join For FreeAn exhaustive test-suite comprises many test cases that test different features on various combinations of browsers, platforms, and devices. Though it is recommended not to skip tests, there are cases where you may want to stop test suite after n test failures, (n is the failure threshold value) number of the test fails while performing Selenium test automation.
The option comes handy in scenarios where there are inter-dependent test cases and failure of one test case causes its dependent test cases to fail. This also avoids the execution of unnecessary Selenium test automation cases that are bound to fail (due to failure of dependent tests).
In this Python tutorial, I’ll explore how to stop test suite after n test failures in pytest using the @pytest.mark.incremental decorator and maxfail command-line option.
Using @pytest.mark.incremental decorator to Stop Test Suite After N Test Failures
The @pytest.mark.incremental decorator is used for skip test in Python with pytest. It is used for marking all the tests that are expected to fail so that the dependent tests are not attempted for execution. It is also used to stop test suite after n test failures in pytest For instance, there are two tests in a class i.e. Test_1 & Test_2 and Test_2 is dependent on Test_1. If Test_1 fails, Test_2 need not be executed unnecessarily as it is bound to fail and its trackback adds no insight.
PyTest provides hook implementations that work together to stop incremental-marked tests in a class. Here is the conftest.py file to introduce the incremental marker used to stop test suite after n test failures.
conftest.py (to skip test in Python with pytest)
import pytest
from selenium import webdriver
import urllib3
import warnings
def pytest_runtest_makereport(item, call):
if "incremental" in item.keywords:
if call.excinfo is not None:
parent = item.parent
parent._previousfailed = item
def pytest_runtest_setup(item):
previousfailed = getattr(item.parent, "_previousfailed", None)
if previousfailed is not None:
pytest.xfail("previous test failed (%s)" % previousfailed.name)
The @pytest.mark.incremental decorator is used in cases where there is a dependency between tests and if a particular test (in a class) fails, the subsequent tests are marked as expected to fail (or marked as xfail) to skip the test in Python with pytest.
How to Skip Test In Python with Pytest Using @pytest.mark.incremental
To demonstrate the usage of @pytest.mark.incremental to skip the test in Python with pytest, we take an automated browser testing example which contains four test scenarios. Class Test_Scenario_1 consists of three Selenium test automation cases and Test_Scenario_2 consists of one test case.
Fixture with scope as the class is used to instantiate the Chrome browser instance. Implementation to skip tests in Python with a pytest is also included in conftest.py.
Conftest.py
xxxxxxxxxx
import pytest
from selenium import webdriver
import urllib3
import warnings
def pytest_runtest_makereport(item, call):
if "incremental" in item.keywords:
if call.excinfo is not None:
parent = item.parent
parent._previousfailed = item
def pytest_runtest_setup(item):
previousfailed = getattr(item.parent, "_previousfailed", None)
if previousfailed is not None:
pytest.xfail("previous test failed (%s)" % previousfailed.name)
.fixture(scope="class")
def driver_init(request):
web_driver = webdriver.Chrome()
request.cls.driver = web_driver
yield
web_driver.close()
# web_driver.quit()
test_step.py
xxxxxxxxxx
#Using @pytest.mark.incremental decorator to stop test suite after n test failure using Selenium test automation in Python with pytest
import pytest
import pytest_html
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
from time import sleep
import sys
import urllib3
import warnings
# @pytest.mark.incremental Stop Test Suite after N Test Failures in Pytest
.mark.usefixtures("driver_init")
.mark.incremental
class Test_Scenario_1:
def test_1(self):
self.driver.get('https://www.lambdatest.com/blog/')
self.driver.maximize_window()
expected_title = "LambdaTest | A Cross Browser Testing Blog"
assert expected_title == self.driver.title
time.sleep(5)
def test_2(self):
self.driver.get('https://www.google.com/')
self.driver.maximize_window()
title = "Google"
assert title == self.driver.title
search_text = "LambdaTest"
search_box = self.driver.find_element_by_xpath("//input[@name='q']")
search_box.send_keys(search_text)
time.sleep(5)
search_box.submit()
time.sleep(5)
# Click on the LambdaTest HomePage Link
# This test will fail as the titles will not match
title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
lt_link = self.driver.find_element_by_xpath("//h3[.='LambdaTest: Cross Browser Testing Tools | Free Automated ...']")
lt_link.click()
time.sleep(10)
assert title == self.driver.title
time.sleep(2)
# As test_2 fails, test_3 will xfail i.e. skipped and all subsequent tests in the same class
# will be marked as xfailed
def test_3(self):
self.driver.get('https://www.lambdatest.com/')
self.driver.maximize_window()
expected_title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest"
assert expected_title == self.driver.title
time.sleep(5)
.mark.usefixtures("driver_init")
.mark.incremental
class Test_Scenario_2:
def test_4(self):
self.driver.get('https://lambdatest.github.io/sample-todo-app/')
self.driver.maximize_window()
self.driver.find_element_by_name("li1").click()
self.driver.find_element_by_name("li2").click()
title = "Sample page - lambdatest.com"
assert title == self.driver.title
sample_text = "Happy Testing at LambdaTest"
email_text_field = self.driver.find_element_by_id("sampletodotext")
email_text_field.send_keys(sample_text)
time.sleep(5)
self.driver.find_element_by_id("addbutton").click()
time.sleep(5)
assert self.driver.find_element_by_xpath("//span[.='Happy Testing at LambdaTest']").text == sample_text
As seen in the implementation, incremental testing is applied to class Test_Scenario_2 and Test_Scenario_2 via the @pytest.mark.incremental decorator.
xxxxxxxxxx
......................................
......................................
from time import sleep
import sys
import urllib3
import warnings
......................................
......................................
.mark.usefixtures("driver_init")
.mark.incremental
class Test_Scenario_1:
..................
..................
def test_2(self):
self.driver.get('https://www.google.com/')
self.driver.maximize_window()
title = "Google"
assert title == self.driver.title
search_text = "LambdaTest"
search_box = self.driver.find_element_by_xpath("//input[@name='q']")
search_box.send_keys(search_text)
time.sleep(5)
search_box.submit()
time.sleep(5)
# Click on the LambdaTest HomePage Link
# This test will fail as the titles will not match
title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
..................
..................
.mark.usefixtures("driver_init")
.mark.incremental
class Test_Scenario_2:
..................
..................
In test case test_2(), Google search for LambdaTest is performed. Once the search results are ready, a click on the first search result is done and the page title is compared with the expected title. For testing, we have deliberately introduced an error in the code and the Selenium test automation case fails as the titles do not match.
xxxxxxxxxx
class Test_Scenario_1:
..................
..................
def test_2(self):
self.driver.get('https://www.google.com/')
.................
..................
search_text = "LambdaTest"
search_box = self.driver.find_element_by_xpath("//input[@name='q']")
search_box.send_keys(search_text)
time.sleep(5)
search_box.submit()
time.sleep(5)
# Click on the LambdaTest HomePage Link
# This test will fail as the titles will not match
title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
..................
..................
# As test_2 fails, test_3 will xfail i.e. skipped and all subsequent tests in the same class
# will be marked as xfailed
def test_3(self):
..................
..................
.mark.usefixtures("driver_init")
.mark.incremental
class Test_Scenario_2:
def test_4(self):
..................
..................
Test case test_3() which follows after test_2() will be skipped from execution and marked as xfail as the previous test [i.e. test_2()] in the same class has failed. Selenium test automation case test_4() which is a part of class Test_Scenario_2() is also marked for incremental testing.
Here is the command for execution:
py.test -rx --verbose --capture=no
Shown below is the execution snapshot:
Here is the final status of the tests performed in the example:
Test_2 executes and fails whereas test_3 execution is skipped due to failure of test_2. Hence, it is marked as xfail. Though the test_4 () passes here, even if it were to fail, the test would be marked as fail (and not xfail) as there is no dependency on the test from the other class.
In case, incremental testing is removed from the above implementation, all the tests would be executed. Only test_2() will fail whereas the remaining three tests would pass as the @pytest.mark.incremental decorator is removed from the implementation. At times you’d have to run multiple test cases in pytest from a single file, read this article to know more.
Using Maxfail Command Line Option to Stop Test Suite After N Test Failures
PyTest offers a command-line option called maxfail which is used to stop test suite after n test failures. For example, to stop the test suite after n test failures, where the value of n is 1. Once a failure is encountered, all the subsequent tests are skipped.
Though it is not a good practice to develop inter-dependent Selenium test automation cases, in some scenarios, you may have to develop such tests. The maxfail option comes handy in such cases as unnecessary execution of dependent tests can be avoided. Shown below is the syntax of maxfail:
py.test --maxfail=<num>
How to Skip Test in Python With pytest Using Maxfail
To demonstrate the usage of maxfail, we take the automated browser testing example which was shown in the section How To Skip Test In Python With pytest Using @pytest.mark.incremental. The implementation-specific to skip test in Python with pytest will be removed, the remaining code remains the same.
conftest.py
xxxxxxxxxx
import pytest
from selenium import webdriver
import urllib3
import warnings
def pytest_runtest_setup(item):
previousfailed = getattr(item.parent, "_previousfailed", None)
if previousfailed is not None:
pytest.xfail("previous test failed (%s)" % previousfailed.name)
.fixture(scope="class")
def driver_init(request):
web_driver = webdriver.Chrome()
request.cls.driver = web_driver
yield
web_driver.close()
# web_driver.quit()
test_step.py
xxxxxxxxxx
#Using maxfail command-line operation to stop test suite after n test failure using Selenium test automation in Python with pytest
import pytest
import pytest_html
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
from time import sleep
import sys
import urllib3
import warnings
.mark.usefixtures("driver_init")
# @pytest.mark.incremental Stop Test Suite after N Test Failures in Pytest
class Test_Scenario_1:
def test_1(self):
self.driver.get('https://www.lambdatest.com/blog/')
self.driver.maximize_window()
expected_title = "LambdaTest | A Cross Browser Testing Blog"
assert expected_title == self.driver.title
time.sleep(5)
def test_2(self):
self.driver.get('https://www.google.com/')
self.driver.maximize_window()
title = "Google"
assert title == self.driver.title
search_text = "LambdaTest"
search_box = self.driver.find_element_by_xpath("//input[@name='q']")
search_box.send_keys(search_text)
time.sleep(5)
search_box.submit()
time.sleep(5)
# Click on the LambdaTest HomePage Link
# This test will fail as the titles will not match
title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
lt_link = self.driver.find_element_by_xpath("//h3[.='LambdaTest: Cross Browser Testing Tools | Free Automated ...']")
lt_link.click()
time.sleep(10)
assert title == self.driver.title
time.sleep(2)
def test_3(self):
self.driver.get('https://www.lambdatest.com/')
self.driver.maximize_window()
expected_title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest"
assert expected_title == self.driver.title
time.sleep(5)
.mark.usefixtures("driver_init")
# @pytest.mark.incremental
class Test_Scenario_2:
def test_4(self):
self.driver.get('https://lambdatest.github.io/sample-todo-app/')
self.driver.maximize_window()
self.driver.find_element_by_name("li1").click()
self.driver.find_element_by_name("li2").click()
title = "Sample page - lambdatest.com"
assert title == self.driver.title
sample_text = "Happy Testing at LambdaTest"
email_text_field = self.driver.find_element_by_id("sampletodotext")
email_text_field.send_keys(sample_text)
time.sleep(5)
self.driver.find_element_by_id("addbutton").click()
time.sleep(5)
assert self.driver.find_element_by_xpath("//span[.='Happy Testing at LambdaTest']").text == sample_text
The example shown below is executed where we stop the test suite after n test failures and the maximum number of failures set to 2. Here is the execution command:
py.test -rx --verbose --capture=no --maxfail=2
As incremental testing is not enabled, all the four Selenium test automation cases will be executed and the execution stops once the number of failures becomes 2. Test case test_2() fails as the window title does not match the expected one, rest all the three test cases are executed. Out of 4 test cases, 1 test case fails and 3 test cases pass. Here is the execution snapshot:
All In All
In this Python tutorial, we looked at how to stop test suite after n test failures. Skip tests in Python with a pytest can be followed in Selenium test automation scenarios where tests are inter-dependent and execution of dependent tests needs to be skipped in case the primary test case fails.
The @pytest.mark.incremental decorator is used to skip tests in Python with PyTest. The other mechanism to stop test suite after n test failures by using the maxfail command-line option. Refer to this article, to learn more about Python fixtures.
This brings an end to this Python tutorial. I hope this article was informative, and you now know skip tests in python or how to stop test suite after n test failures. In case you still have any doubts, do reach out to us in the comment section below, and we’d be happy to help you with your problem. Happy Testing!
Published at DZone with permission of Himanshu Sheth. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments