A Guide To Visual Regression Testing With PlayWright and How to Get Started

I’m pretty sure that you’ve had a situation where you deployed a major UX change on your web app and missed the most obvious issues, like a misaligned button or distorted images. 

Unintended changes on your site can cause not only a sharp decline in user satisfaction but also a large fall in sales and customer retention. By identifying and resolving these discrepancies before the update went live, you could have prevented these outcomes.

This is where visual regression testing comes in, helping you validate visual elements on your web app. Having a visual regression testing strategy prevents issues that could disrupt the user experience and cause users to bounce. 

In this article, we’re focusing on visual regression testing with PlayWright. We’ll break down the concept, go through the benefits of visual testing, and see how to implement it into your testing strategy. Finally, I’ll show you how to combine Checkly and PlayWright for effective visual regression testing.

What Is Visual Regression Testing With PlayWright? 

Visual regression testing with PlayWright is the most reliable way to ensure your web app looks and behaves correctly. It does this by comparing two snapshots of an area of your choice and immediately detecting issues related to the layout, content, or design of your web app. 

When web apps are updated iteratively with new features or optimizations, there is a chance that unintentional visual changes will occur. If these changes go unnoticed, they can negatively affect the user experience, cause customer annoyance, or even lower engagement or sales.

The main goal of visual regression testing is to provide predictable and consistent visual behavior over iterations, avoiding regressions and improving the application's overall quality.

Visual Regression Testing Key Concepts 

To understand the concept of visual regression testing, let’s look at these key concepts related to it: 

• Image comparison: To identify changes between older and more recent iterations of web pages, visual regression testing uses image comparison techniques. To draw attention to any visual changes or discrepancies, PlayWright takes screenshots and applies picture-diffing methods to them.
• Baseline establishment: Setting up a baseline entails establishing a point of reference for the visual design of the application's initial release. This baseline is used as a benchmark for comparison in later testing iterations.
• Automated workflows: With PlayWright, automation is essential to visual regression testing since it makes it possible to execute tests repeatedly and smoothly across a variety of devices and browsers.
• Version control system integration: By incorporating visual regression tests into version control systems, development teams can work together more easily, and traceability is guaranteed.
  
  Check how the feature compares snapshots here:

Benefits of Visual Regression Testing

Now, let’s look at the benefits and advantages of visual regression testing: 

• Consistent UI/UX stability: Visual regression testing ensures a consistent and stable user interface and experience across different versions, guaranteeing reliability for end-users.
• Efficiency and cost-effectiveness: Automating visual regression tests using PlayWright saves time and resources by reducing the need for manual checks, leading to more efficient testing processes.
• Early identification of issues: Detecting visual defects early in the development cycle allows for swift issue resolution, minimizing the chances of releasing flawed features and enhancing overall software quality.
• Cross-platform compatibility: With PlayWright, visual regression testing verifies visual elements across multiple browsers and devices, ensuring uniformity and compatibility in diverse environments.
• Confidence in deployments: Regular visual regression testing instills confidence in software releases, decreasing the likelihood of unexpected visual regressions when rolling out updates or new features.

What Does Visual Regression Testing Not Detect? 

Visual testing, while crucial for assessing the graphical interface and layout of web applications, does not cover certain aspects related to the functionality and underlying code. Some areas that visual testing does not encompass include:

• Functional testing: This testing verifies that the application operates in accordance with its specifications, ensuring that each feature, component, or interaction functions correctly and performs as intended, encompassing form submissions, user actions, data processing, and other essential operations.
• Security testing: Assessing security measures within the application is vital. Security testing identifies vulnerabilities, weaknesses, and potential threats, aiming to prevent data breaches, unauthorized access, or any form of security compromise. It involves examining authentication methods, data protection mechanisms, encryption, and defenses against diverse cyber threats.
• Accessibility testing: Ensuring the application's accessibility to all user groups, including individuals with disabilities, is crucial. Accessibility testing confirms compliance with accessibility standards like WCAG, focusing on features such as compatibility with screen readers, keyboard navigation, contrast adjustments, and other elements to ensure an inclusive user experience for diverse audiences.
• API and backend testing: API and backend testing involve evaluating the functionality and responses of the application's backend components, such as APIs, databases, and server-side operations. Unlike visual testing, which focuses on the front end, this testing requires distinct methodologies to directly interact with and assess the backend systems, ensuring their proper functioning and accuracy in handling data and operations.

Getting Started With Visual Regression Testing

Checkly natively supports PlayWright Test Runner for browser checks, so you can now use its visual snapshot testing feature with Checkly.

These are two important assertions to get you started: 

• .toHaveScreenshot() will help you visually compare a screenshot of your page to a golden image/reference snapshot
• .toMatchSnapshot() will compare any string or Buffer value to a golden image/reference snapshot

Before getting started, make sure you’ve downloaded the newest version of Checkly: 

• Checkly CLI v4.4.0 or later.
• Checkly Agent v3.2.0 or later.

Step 1

Add expect(page).toHaveScreenshot() to your browser check script. Here’s an example: 

Step 2

Run your browser check. The first time you run it, you will get an error indicating that no golden image/reference snapshot exists yet.

A snapshot doesn't exist at /tmp/19g67loplhq0j/script.spec.js-snapshots/Playwright-homepage-1-chromium-linux.png.

Step 3

Generate a golden image snapshot by clicking the “Run script and update golden image” option in the “Run script” button.

This step will generate a golden image. You can check your golden image in the “Golden Files” tab in the editor. You can now save your check, and on each check run, the golden image will be compared to the actual screenshot.

When your check fails due to a visual difference, you will see the difference between the golden image and the actual screenshot in your check result.

To find out how to configure visual regression testing with Checkly, please check our docs. 

Conclusion

Mastering the intricacies of visual regression testing and API testing is paramount for delivering resilient and high-performing software applications. While visual testing ensures the visual integrity of your application, API testing guarantees the reliability of backend functionalities. To enhance your testing strategy and safeguard against regressions, consider combining PlayWright with other monitoring tools like Checkly.

More Resources

• The official PlayWright guide on visual comparison and snapshot testing
• The .toHaveScreenshot() API reference
• The .toMatchSnapshot() API reference