How To Modify HTTP Request Headers in Java Using Selenium Webdriver

One of the most common test automation challenges is how do we modify the request headers in Selenium WebDriver. As an automation tester, you would come across this challenge for any programming language, including Java. Before coming to a solution, we need to understand the problem statement better and arrive at different possibilities to modify the header request in Java while working with Selenium WebDriver Tutorial.

In this Selenium Java tutorial, we will learn how to modify HTTP request headers in Java using Selenium WebDriver with different available options.

Starting your journey with Selenium WebDriver? Check out this step-by-step guide to perform Automation testing using Selenium WebDriver.

So let’s get started!

What Are HTTP Headers

HTTP headers are an important part of the HTTP protocol. They define an HTTP message (request or response) and allow the client and server to exchange optional metadata with the message. They are composed of a case-insensitive header field name followed by a colon, then a header field value. Header fields can be extended over multiple lines by preceding each extra line with at least one space or horizontal tab.

Headers can be grouped according to their contexts:

1. Request Headers: HTTP request headers are used to supply additional information about the resource being fetched and the client making the request.
2. Response Headers: HTTP response headers provide information about the response. The Location header specifies the location of a resource, and the server header presents information about the server providing the resource.
3. Representation Headers: HTTP representation headers are an important part of any HTTP response. They provide information about protocol elements like mime types, character encodings, and more. This makes them a vital part of processing resources over the internet.
4. Payload Headers: HTTP payload headers contain data about the payload of an HTTP message (such as its length and encoding) but are representation-independent.Deep Dive Into HTTP Request Headers
   The HTTP Request header is a communication mechanism that enables browsers or clients to request specific webpages or data from a (Web) server. When used in web communications or internet browsing, the HTTP Request Header enables browsers and clients to communicate with the appropriate Web server by sending requests.
   The HTTP request headers describe the request sent by the web browser to load a page. It’s also referred to as the client-to-server protocol. The header includes details of the client’s request, such as the type of browser and operating system used by the user and other parameters required for the proper display of the requested content on the screen.
   Here is the major information included within the HTTP request headers:

• IP address (source) and port number.
• URL of the requested web page.
• Web Server or the destination website (host).
• Data type that the browser will accept (text, html, xml, etc.).
• Browser type (Mozilla, Chrome, IE) to send compatible data.

In response, an HTTP response header containing the requested data is sent back by the.

The Need to Change the HTTP Request Headers

Can you guess why we even need to change the request header once it is already set into the scripts?

Here are some of the scenarios where you might need to change the HTTP Request Headers:

1. Testing the control and/or testing the different variants by establishing appropriate HTTP headers.
2. The need to test the cases when different aspects of the web application or even the server logic have to be thoroughly tested.
3. Since the HTTP request headers come to use to enable some specific parts of a web application logic, which in general would be disabled in a normal mode, modification of the HTTP request headers may be required from time to time per the test scenario.

Testing the guest mode on a web application under test is the ideal case where you might need to modify the HTTP request headers.

However, the function of modifying the HTTP request header, which Selenium RC once supported, is now not handled by Selenium Webdriver.

This is why the question arises about how we change the header request when the test automation project is written using the Selenium framework and Java.

How To Modify Header Requests in Selenium Java Project

In this part of the Selenium Java tutorial, we look at the numerous ways to modify header requests in Java. Broadly, there are a few possibilities, following which one can modify the header request in the Java-Selenium project.

1. Using a driver/library like REST Assured instead of Selenium.
2. Using a reverse proxy such as browser mob-proxy or some other proxy mechanism.
3. Using a Firefox browser extension, which would help to modify the headers for the request.

Let us explore each possibility one by one:

Modify HTTP Request Headers Using REST Assured Library

Along with Selenium, we can make use of REST Assured, which is a wonderful tool to work with REST services in a simple way.

The prerequisites to configure REST Assured with your project in any IDE (e.g., Eclipse) is fairly easy. After setting up Java, Eclipse, and TestNG, you would need to download the required REST Assured jar files.

After the jar files are downloaded, you have to create a project in Eclipse and add the downloaded jar files as external jars to the Properties section. This is again similar to the manner in which we add Selenium jar files to the project. Once you have successfully set up the Java project with the REST Assured library, you are good to go.

We intend to create a mechanism so that the request header is customizable. To achieve this with the possibility mentioned above, we first need to know the conventional way to create a request header.

Let’s consider the following scenario:

• We have one Java class named RequestHeaderChangeDemo where we maintain the basic configurations
• We have a test step file named TestSteps, where we will call the methods from the RequestHeaderChangeDemo Java class through which we will execute our test.

Observe the below Java class named RequestHeaderChangeDemo.

The BASE_URL is the Amazon website on which the following four methods are applied:

• authenticateUser
• getProducts
• addProducts
• removeProduct

public class RequestHeaderChangeDemo
{
    private static final String BASE_URL = "https://amazon.com";
 
    public static IRestResponse<Token> authenticateUser(AuthorizationRequest authRequest) {
 
    RestAssured.baseURI = BASE_URL;
    RequestSpecification request = RestAssured.given(); 
    request.header("Content-Type", "application/json");
 
    Response response = request.body(authRequest).post(Route.generateToken());
    return new RestResponse(Token.class, response);
    }
 
    public static IRestResponse<Products> getProducts() 
    {
 
    RestAssured.baseURI = BASE_URL;
    RequestSpecification request = RestAssured.given(); 
    request.header("Content-Type", "application/json");
    Response response = request.get(Route.products());
    return new RestResponse(Products.class, response);
    }
 
    public static IRestResponse<UserAccount> addProduct(AddProductsRequest addProductsRequest, String token) 
    {
    RestAssured.baseURI = BASE_URL;
    RequestSpecification request = RestAssured.given();
    request.header("Authorization", "Bearer " + token)
    .header("Content-Type", "application/json");
 
    Response response = request.body(addProductsRequest).post(Route.products());
    return new RestResponse(UserAccount.class, response);
    }
 
    public static Response removeProduct(RemoveProductRequest removeProductRequest, String token)
    {
 
    RestAssured.baseURI = BASE_URL;
    RequestSpecification request = RestAssured.given();
    request.header("Authorization", "Bearer " + token)
    .header("Content-Type", "application/json");
 
    return request.body(removeProductRequest).delete(Route.product());,
    }
}

In the above Java class file, we have repeatedly sent the BASE_URL and headers in every consecutive method. Example is shown below:

RestAssured.baseURI = BASE_URL;
RequestSpecification request = RestAssured.given(); 
request.header("Content-Type", "application/json");
Response response = request.body(authRequest).post(Route.generateToken());

The request.header method requests the header in the JSON format. There is a significant amount of duplication of code which reduces the maintainability aspect of the code.

This can be avoided if we initialize the RequestSpecification object in the constructor and make these methods non-static (i.e. creating the instance method).

Since the instance method in Java belongs to the Object of the class and not to the class itself, the method can be called even after creating the Object of the class. Along with this, we will also override the instance method.

Converting the method to an instance method results in the following advantages:

• Authentication is done only once in one RequestSpecification Object. There won’t be any further need to create the same for other requests.
• Flexibility to modify the request header in the project.

Therefore, let us see how both the Java class RequestHeaderChangeDemo and the test step file TestSteps look when we use the instance method.

Java Class for class RequestHeaderChangeDemo with instance method

public class RequestHeaderChangeDemo
{
    private final RequestSpecification request;
    public RequestHeaderChangeDemo(String baseUrl) 
    {
        RestAssured.baseURI = baseUrl;
        request = RestAssured.given();
        request.header("Content-Type", "application/json");
    }
 
    public void authenticateUser(AuthorizationRequest authRequest) 
    {
        Response response = request.body(authRequest).post(Route.generateToken());
        if (response.statusCode() != HttpStatus.SC_OK)
        throw new RuntimeException("Authentication Failed. Content of failed Response: " +             response.toString() + " , Status Code : " + response.statusCode());
        
        Token tokenResponse = response.body().jsonPath().getObject("$", Token.class);
        request.header("Authorization", "Bearer " + tokenResponse.token);
    }
 
    public IRestResponse<Products> getProducts() 
    {
        Response response = request.get(Route.products());
        return new RestResponse(Products.class, response);
    }
 
    public IRestResponse<UserAccount> addProduct(AddProductsRequest addProductsRequest) 
    {
        Response response = request.body(addProductsRequest).post(Route.products());
        return new RestResponse(UserAccount.class, response);
    }
 
    public Response removeProducts(RemoveProductRequest removeProductRequest)
    {
        return request.body(removeProductRequest).delete(Route.product());
    }
}

Code Walkthrough

1. We have created a constructor to initialize the RequestSpecification object containing BaseURL and Request Headers.
2. Earlier, we had to pass the token in every request header. Now, we put the tokenresponse into the same instance of the request as soon as we receive it in the method authenticateUser(). This enables the test step execution to move forward without adding the token for every request like it was done earlier. This makes the header available for the subsequent calls to the server.
3. This RequestHeaderChangeDemo Java class will now be initialized in the TestSteps file.

We change the TestSteps file in line with the changes in the RequestHeaderChangeDemo Java class.

public class TestSteps
{
    private final String USER_ID = " (Enter the user id from your test case )";    
    private Response response;
    private IRestResponse<UserAccount> userAccountResponse;
    private Product product;
    private final String BaseUrl = "https://amazon.com";
    private RequestHeaderChangeDemo endPoints;
    
    @Given("^User is authorized$")
    public void authorizedUser()
    {
        endPoints = new RequestHeaderChangeDemo (BaseUrl);
        AuthorizationRequest authRequest = new AuthorizationRequest("(Username)", "(Password)");
        endPoints.authenticateUser(authRequest);
    }
 
    @Given("^Available Product List$")
    public void availableProductLists() 
    {       
        IRestResponse<Products> productsResponse = endPoints.getProducts();
        Product = productsResponse.getBody().products.get(0);
    }
 
    @When("^Adding the Product in Wishlist$")
    public void addProductInWishList() 
    {
        ADDPROD code = new ADDPROD(product.code);
        AddProductsRequest addProductsRequest = new AddProductsRequest(USER_ID, code);
        userAccountResponse = endPoints.addProduct(addProductsRequest);
    }
 
    @Then("^The productis added$")
    public void productIsAdded() 
    {      
        Assert.assertTrue(userAccountResponse.isSuccessful());
        Assert.assertEquals(201, userAccountResponse.getStatusCode());
        Assert.assertEquals(USER_ID, userAccountResponse.getBody().userID);
        Asert.assertEquals(product.code, userAccountResponse.getBody().products.get(0).code);
    }
 
    @When("^Product to be removed from the list$")
    public void removeProductFromList() 
    {
        RemoveProductRequest removeProductRequest = new RemoveProductRequest(USER_ID, product.code);
        response = endPoints.removeProduct(removeProductRequest);
    }
 
    @Then("^Product is removed$")
    public void productIsRemoved() 
    {
        Assert.assertEquals(204, response.getStatusCode());
        userAccountResponse = endPoints.getUserAccount(USER_ID);
        Assert.assertEquals(200, userAccountResponse.getStatusCode());     
        Assert.assertEquals(0, userAccountResponse.getBody().products.size());
    }
}

Code Walkthrough

Here’s what we have done in the modified implementation:

1. Initiatialized RequestHeaderChangeDemo class objects as endpoints.
2. The BaseURL was passed in the first method (i.e. authorizedUser).
3. Within the method authorizedUser, we invoked the constructor authenticateUser of the RequestHeaderChangeDemo class.
4. Hence the same endpoint object is used by the subsequent step definitions.

Modify HTTP Request Headers Using Reverse Proxy Like Browser Mob-Proxy

As the name suggests, we can opt for using proxies when dealing with the request header changes in a Java-Selenium automation test suite. As Selenium forbids injecting information amidst the browser and the server, proxies can come to a rescue.

This approach is not preferred if the testing is being performed behind a corporate firewall.

Being a web infrastructure component, Proxy makes the web traffic move through it by positioning itself between the client and the server. In the corporate world, proxies work similarly, making the traffic pass through it, allowing the ones that are safe and blocking the potential threats. Proxies come with the capability to modify both the requests and the responses, either partially or completely.

The core idea is to send the authorization headers, bypassing the phase that includes the credential dialogue, also known as the basic authentication dialog. However, this turns out to be a tiring process, especially if the test cases demand frequent reconfigurations.

This is where the browser mob-proxy library comes into the picture. When you make the proxy configuration part of the Selenium automation testing suite, the proxy configuration will stand valid each time you execute the test suite.

Let us see how we can use the browser mob-proxy with a sample website that is secured with basic authentication. To tackle this, we might narrow down two possible ways:

1. Add authorization headers to all requests with no condition or exception.
2. Add headers only to the requests which meet certain conditions.

Though we will not address headers management problems, we would still demonstrate how to address authorization issues with the help of the browser mob-proxy authorization toolset.

In this part of the Selenium Java tutorial, we will focus only on the first methodology (i.e. adding authorization headers to all the requests).

First, we add the dependencies of browsermob-proxy in pom.xml

.......................
    .......................
    <dependencies>
        <dependency>
            <groupId>net.lightbody.bmp</groupId>
            <artifactId>browsermob-core</artifactId>
            <version>2.1.5</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    .......................
    .......................

public class caseFirstTest
{
    WebDriver driver;
    BrowserMobProxy proxy;
 
    @BeforeAll
    public static void globalSetup()
    {
        System.setProperty("webdriver.gecko.driver", "(path of the driver)");
    }
 
    @BeforeEach
    public void setUp()
    {
        setUpProxy();
        FirefoxOptions Options = new FirefoxOptions();
        Options.setProxy(ClientUtil.createSeleniumProxy(proxy));
        driver = new FirefoxDriver(Options);
    }
 
    @Test
    public void testBasicAuth()
    {
        driver.get("https://webelement.click/stand/basic?lang=en");
        Wait<WebDriver> waiter = new FluentWait(driver).withTimeout(Duration.ofSeconds(50)).ignoring(NoSuchElementException.class);
        String greetings = waiter.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("(Mention the xpath)"))).getText();
        Assertions.assertEquals("(message");
    }
 
    @AfterEach
    public void tearDown()
    {
        if(driver != null)
        {
            driver.quit();
        }
        if(proxy != null)
        {
            proxy.stop();
        }
    }
    private void setUpProxy(
    {
    }
}

If you want to pass this approach to all the header requests, a particular proxy, in this case, the forAllProxy method should be invoked as shown below:

public void forAllProxy()
{
    proxy = new BrowserMobProxyServer();
    try {
        String authHeader = "Basic " + Base64.getEncoder().encodeToString("webelement:click".getBytes("utf-8"));
        proxy.addHeader("checkauth", authfirstHeader);
    }
    catch (UnsupportedEncodingException e)
    {
        System.err.println("the Authorization can not be passed");
        e.printStackTrace();
    }
    proxy.start(0);
}

In the above code, the line that starts with String authHeader states that we are creating the header, and this will be added to the requests. After that, these requests are passed through the proxy we created in proxy.addHeader(“checkauth”, authfirstHeader).

try {
        String authHeader = "Basic " + Base64.getEncoder().encodeToString("webelement:click".getBytes("utf-8"));
        proxy.addHeader("checkauth", authfirstHeader);
    }
    catch (UnsupportedEncodingException e)
    {
	………………………………………………………………………………
	………………………………………………………………………………
	……………………………………………………………………………...
    }
    proxy.start(0);
}

Eventually, we start the proxy setting 0 to mark the start parameter, and the proxy starts on the port.

Modify HTTP Request Headers Using Firefox Extension

In this part of the Selenium Java tutorial, we look at how to modify the header requests using the appropriate Firefox browser extension. The major drawback of this option is that it works only with Firefox (and not other browsers like Chrome, Edge, etc.).

Perform the following steps to modify HTTP request headers using a Firefox extension:

• Download the Firefox browser Extension
• Load the extension.
• Set up the extension preferences.
• Set the Desired Capabilities.
• Prepare the test automation script.

Let us go through each step one by one:

1. Download the Firefox browser extension

Search for the firefox extension with .*xpi and set it up in the project

2. Load the Firefox extension

Add the Firefox profile referring to the below code:

FirefoxProfile profile = new FirefoxProfile();
File modifyHeaders = new File(System.getProperty("user.dir") + "/resources/modify_headers.xpi");
profile.setEnableNativeEvents(false); 
 
try {
    profile.addExtension(modifyHeaders); 
}
catch (IOException e)
{
    e.printStackTrace();
}

3. Set the extension preferences

Once we load the Firefox extension into the project, we set the preferences (i.e. various inputs that need to be set before the extension is triggered). This is done using the profile.setPreference method.

This method sets the preference for any given profile through the key-set parameter mechanism. Here the first parameter is the key that sets the value in addition to the second parameter, which sets a corresponding integer value.

Here is the reference implementation:

profile.setPreference("modifyheaders.headers.count", 1);
profile.setPreference("modifyheaders.headers.action0", "Add");
profile.setPreference("modifyheaders.headers.name0", "Value");
profile.setPreference("modifyheaders.headers.value0", "numeric value");
profile.setPreference("modifyheaders.headers.enabled0", true);
profile.setPreference("modifyheaders.config.active", true);
profile.setPreference("modifyheaders.config.alwaysOn", true);

In the above code, we list the number of times we want to set the header instance. 

profile.setPreference("modifyheaders.headers.count", 1);

Next, we specify the action, and the header name and header value contain the dynamically received values from the API calls.

profile.setPreference("modifyheaders.headers.action0", "Add");

For the rest of the line of the implementation of .setPreference, we enable all so that it allows the extension to be loaded when the WebDriver instantiates the Firefox browser along with setting the extension in active mode with HTTP header.

4. Set up the Desired Capabilities

The Desired Capabilities in Selenium are used to set the browser, browser version, and platform type on which the automation test needs to be performed.

Here we how we can set the desired capabilities:

DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setBrowserName("firefox");
capabilities.setPlatform(org.openqa.selenium.Platform.ANY);
capabilities.setCapability(FirefoxDriver.PROFILE, profile);
 
WebDriver driver = new FirefoxDriver(capabilities);
driver.get("url");

What if you want to modify HTTP request headers with a Firefox version that is not installed on your local (or test) machine? This is where LambdaTest, the largest cloud-based automation testing platform that offers faster cross browser testing infrastructure, comes to the rescue.

With LambdaTest, you have the flexibility to modify HTTP request headers for different browsers and platform combinations. If you are willing to modify HTTP request headers using the Firefox extension, you can use LambdaTest to realize the same on different versions of the Firefox browser.

5. Draft the entire test automation script

Once you have been through all the above steps, we proceed with designing the entire test automation script:

public void startwebsite()
{
    FirefoxProfile profile = new FirefoxProfile();
    File modifyHeaders = new File(System.getProperty("user.dir") + "/resources/modify_headers.xpi");
    profile.setEnableNativeEvents(false); 
    try
    {
        profile.addExtension(modifyHeaders); 
    }
    catch (IOException e)
    {
        e.printStackTrace(); 
    }
 
    profile.setPreference("modifyheaders.headers.count", 1);
    profile.setPreference("modifyheaders.headers.action0", "Add");
    profile.setPreference("modifyheaders.headers.name0", "Value");
    profile.setPreference("modifyheaders.headers.value0", "Numeric Value");
    profile.setPreference("modifyheaders.headers.enabled0", true);
    profile.setPreference("modifyheaders.config.active", true);
    profile.setPreference("modifyheaders.config.alwaysOn", true);
 
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.setBrowserName("firefox");
    capabilities.setPlatform(org.openqa.selenium.Platform.ANY);
    capabilities.setCapability(FirefoxDriver.PROFILE, profile);
 
    WebDriver driver = new FirefoxDriver(capabilities);
    driver.get("url");
}

Conclusion

In this Selenium Java tutorial, we explored three different ways to handle the modifications on the HTTP request headers. Selenium in itself is a great tool and has consistently worked well in web automation testing.