Things Every Product Manager Must Know About Testing

Being responsible for the development and maintenance of a rapidly evolving tech product is arguably one of the most technical and fast-paced jobs out there. The fact that the whole landscape of relevant technologies is also evolving outside your company and product adds another layer of complexity to the mix.  

New devices, new features on your favorite cloud platform, latest APP frameworks, new DB paradigms, best IDE for your team, etc. The number of right decisions that a product team needs to make to ensure a smooth functioning application is quite large.

Despite variabilities like business and programming paradigms, application testing as a complement to application development has been widely adopted as an effective and indispensable tool in the software development life cycle. Let’s try to cover all the basics briefly: 

Why Is It Important to Test Software? 

Establishing a healthy system for testing, which is running in sync with the design and development part of the application development effort, is very important. It is a scientific way of capturing, documenting, sharing, and resolving problems in a software application in a collaborative manner. It helps prevent issues from causing business loss or poor user experiences. 

For example, in 2019, British Airways had to cancel 100 and delay 200 flights due to issues with their online portal, which caused a crowd of people and many long queues outside Heathrow airport; or how about this report which explains how a financial service provider in the US lost $440 Million because of a software error. Needless to say, no business wants to go through any such tragedy, and software testing is the most effective way to ensure such events don’t occur.

        Visualizing the testing workflow

We can say that software testing, when done correctly, ensures quality and prevents business failure and customer unhappiness. 

Right Strategy and Tools  

How and what to test during the development of a software application depends mainly on the particulars of the business use case. Developing the most efficient and cost-effective testing strategy is not a trivial task. It is easy to get lost in the sea of online advice and waste time, but you can prevent that if you understand the fundamentals we are laying out here: 

Types of Testing

Testing is a broad field of study, and there are many different ways to classify types, but these are the basic layers at which tests are usually implemented:  

1. Unit Testing: Testing parts of the codebase through test code, usually part of the same bundle as the working application code. 

2. Integration Testing involves testing sections of the codebase as interdependent functions or modules through test code or other tools. 

3. System Testing: Testing the workings of an application at the level of features like login, signup, and other supported flows. This helps validate parts of the application working together. 

4. Acceptance Testing: This is usually the final stage of testing in which the fully assembled application with data is tested in a live or pre-production environment. This involves testing with actual or mock users.

5. Performance and Security Testing: With an increase in the number of users, it becomes important to ensure that the servers can handle the request loads at peak usage times. Also, maintaining end-to-end security at each point of the transaction between the app and user.

Modes of Testing

Based on Who Is Testing

Manual Testing: Testing of any component or aspect of a software application carried out by a human. 

Automated Testing: Testing of software components carried out by other applications and tools, with minimum human intervention. 

Based on the Relation Between Tester and Application

Black-Box Testing: When the tester is evaluating an application from outside while not being aware of the internal structure and functions.  

White-Box Testing: When testing is carried out with know-how and consideration for the internal structure of the application.

Testing DevOps

Development Operations or DevOps is a widely used term that refers to a range of activities, including the coding, testing, building, and deployment of software applications. For a product team, the main task is to implement a unit and integration testing plan as part of the technical architecture. Depending on the level of automation, teams use:

• Continuous Integration (CI): This term is used to describe workflows, where the process of validating code commits of individual contributors and merging with the parent repository is carried out in a seamless automated manner. 
• Continuous Delivery/Deployment (CD): This is when a code commit is validated, merged with the parent, and in the same flow, also deployed to a testing or production server. 

Testing in a CI/CD Workflow Usually Involves 

• Writing Unit Tests: This is a default part of all major programming language packages. Read more about unit testing in Javascript, Python, Java/Kotlin, Swift, and Go. With the advent of no-code/low-code and AI-based tools like Devmate, Ponicode, etc., you can now automate parts of the Unit test writing process also.
• Testing Source Code and Build: This involves running unit tests in batches, with other tests on the code to test functionality and code quality. After coding, the next task is to bundle the application into a deployable format as specified by the technical architecture. It involves post-processing the source code and installing packages and dependencies as needed.  

All major cloud hosts like Github, Gitlab, and Bitbucket have native CI/CD support. 

• Testing Backend and Database:  For APIs and backend applications like micro-services, integration testing is a very crucial step. Because of complicated architecture and many dependencies, it is important to ensure proper documentation, performance, and security of REST APIs.  

Also, the data being passed along needs to be tested to ensure proper values, constraints, etc. Testing DB schema, tables, triggers, etc., separately through specially designed queries and tools is sometimes referred to as Database Testing.   

Using tools like Swagger and Apigee will allow you to generate, document, and monitor your API workflows.

• Testing UI: Testing the application user interface in depth is the job of QA teams, but most developers will have a test deployment setup on their dev machine for a brief round of validations that can be performed before code commits. The main goal for this is to ensure proper integration between the database, APIs, dependencies, and the user interface.

Testing Business With Users

Once an application is deployed, the QA team starts work on the system and acceptance level testing. The starting point for this activity is the test plan document which captures in detail the expected behavior of the application during various business scenarios. It highlights all the mission-critical workflows and helps the testing team in understanding the business deeply. With a test plan at hand, the next step is to: 

• Document Test Cases: For a manual scenario, test cases are maintained in a collaborative documentation platform, and the tester is responsible for execution and reporting on the test runs. Tools like JIRA, Trello, etc., are popular choices for this task. 

But in an automation scenario, the test cases are encoded into a script or 3^rd party automation tool like Selenium. 

• Prepare Mock Data: Most test scenarios require users to input information; this behavior is replicated during system testing by preparing and using mock data as a substitute for user data.  

Tools like Visual Studio, Devart, DTM, etc., can help you generate mock data.

• Test Execution: With the right script and data, the test run takes place. A report is generated at the end, which captures various metrics, helping gauge the outcome of the run. Another priority at this step is to execute on an applicable range of devices and browsers to ensure a wide range of compatibility and accessibility.
• Documenting Issues and Exceptions: During and after test execution, all unexpected behavior is captured and documented in the tracking tool, where it is marked up with meta-data like date/time, location, screenshots/videos, steps to recreate, severity, ownership, etc.
• Iterate Until Fixed: The dev team fixes the issues logged in the bug tracker, and the cycle repeats until the QA leads give a green light to the app. 

Role and Perspective of Different Stakeholders

At the level of business leadership, the expectation from the testing process is to ensure that the application(s) is deployed and available to the users while ensuring the best performance, accessibility, and experience. All the documented business requirements and design standards need to be implemented without any discrepancies. Ultimately, the product should be able to fulfill the business goals, e.g., core workflows like sign up, log in, booking, product purchase, etc., should work in users’ hands without any hiccups.

From a technical leadership perspective, including PMs, the main goal is to achieve continuous development and integration flow (CI/CD). The development and QA/QC teams work with each other in a synchronized manner. Agile has been widely accepted as the leading Product Management methodology in today’s industry. The technical leaders will maintain a healthy QAOPS workflow, ensuring that the scripts are running as scheduled and generating relevant reports periodically. Constantly monitoring these reports and keeping a keen eye to ensure that the whole application ecosystem delivers on business goals.

At the level of a programmer, the goal of testing is to check and validate any new additions or changes to the code base before and after they are deployed. In other words, testing for ‘bugs’ in the application code in one’s development, testing/staging, and finally, the production environment. Guidelines for testing within the scope of a particular module are shared with the team, and each contributing member has to write unit tests to cover any new functions or changes they have added to the codebase.  

After the initial sanity check on the development side, the QA/QC team takes on testing it further. The testing process can be manual, automated, or mixed. Testers will document application behavior and anomalies in the form of defects logged in a reporting portal, where they can be assigned for fixing and tracking. 

After QA approves a version of the app, it is released to a controlled set of users for the first stages of acceptance testing. 

Reporting and Communications

Once you have a sound strategy, the right tools and team are in place. You need dashboards, emails, and messages flowing between the stakeholders, acting as a stream of relevant information regarding tests, bug fix status, performance, etc. It usually involves:

• The business team evaluates the current state of application performance, superimposes near future business goals, and communicates the vision to the technical leadership and design team in form of business requirement documents.
• Technical leadership with the QA team updates the test plan to include the overall approach collaboratively, involving all stakeholders. 
• Updating the test cases in the bug tracker, like a traceability matrix. Assigning ownership and passing the tasks to relevant developers.
• Updating manual and automation scripts to include new cases and doing mock runs.
• Reports that gather and display the status and outputs of all the test runs in a time frame, going out as emails, messages, and notifications. 

It is important to have the relevant information reach all parties to ensure that the right action can be taken in the event of a discrepancy. When an application is being worked on regularly, it helps the stakeholders gauge the overall health of the application. 

Conclusion

It is imperative to set up a system where all participants can effectively collaborate on the testing process. To get the best results, you need the whole team to work together.  Armed with the right knowledge and tools, you can effectively improve the quality of deliverables and achieve great things with your product. Although some would argue about the resource-heavy and time-consuming nature of the testing workflow, in a proper development environment, testing provides a unique opportunity to improve the quality of any software application without having to risk any real user or business opportunities.