Top 7 Best Practices DevSecOps Team Must Implement in the CI/CD Process

Almost every organization has implemented CI/CD processes to accelerate software delivery. However, with this increased speed, a new security challenge has emerged. Deployment speed is one thing, but without proper software checks, developers may inadvertently introduce security vulnerabilities, leading to grave risks to business operations. As a result, most organizations are either making DevOps responsible for ensuring the security of their delivery process or creating a dedicated DevSecOps team with the same goal.

In this article, we will discuss the top seven best practices that DevSecOps teams can implement in their CI/CD process to make their software delivery process more secure.

Top Security Challenges Faced by DevSecOps 

The top challenges that DevSecOps teams face when trying to secure the CI/CD process include:

• Diverse DevOps toolchains in CI/CD processes lead to fragmented security-related data. 
• Evaluation of software risks involves processing a lot of data at various stages – build, test, and deploy.
• Manual checks to ensure SDLC compliance with numerous incremental software changes every week are overwhelming the DevSecOps team.
• Vulnerability management is especially challenging because there is no proper mechanism to allow specific low-risk vulnerabilities and keep track of them.
• Auditing an application at regular intervals is very time-consuming because it requires the DevSecOps team to review system logs manually. 

To overcome these challenges, DevSecOps teams must rethink their strategies and implement the necessary tools and best practices to ensure security in the software delivery process. 

1. Integrate Security Testing Into CI/CD Pipelines

The rise of open-source tools and libraries makes applications more vulnerable. Hence, security testing to analyze source code and find security vulnerabilities should be automated. Without proper checks, vulnerabilities can go into production, and applications will be susceptible to attack. A well-known security testing strategy to find source code and binary vulnerabilities is Static/Dynamic Application Security Testing (SAST/DAST), which is offered in scanning solutions such as Sonarqube, Prisma Cloud, HCL AppScan, Jfrog Xray Scanning, and Aquawave. The goal is to seamlessly integrate this scanning technology into the CI/CD pipeline by automatically executing it after the build process is complete. Then, based on the software quality and vulnerabilities, the pipeline proceeds to the next step or fails.  

2. Understand the Risk of Applications and Dependent Services

DevSecOps must empower application owners to understand the security risks of an application and all its services. Holistic information about the security vulnerabilities of each service, deployment date, and the development team will help owners make decisions faster regarding deployments and delivery. Collecting and centralizing this information should be automated, not manual.

3. Track the Delivery Bill of Materials for Each Software Release Going Through the CI/CD Process

The Delivery Bill of Materials (DBOM) is a software supply chain and security management building block. DBOM includes reports related to security risks, quality, performance, and testing, as well as the development and deployment history of the tools used to deliver the application. 

To expedite the decision-making and delivery process, DevSecOps must attempt to centralize the DBOM. However, doing so in larger software development organizations is cumbersome without a purpose-built solution that includes a dashboard that tracks DBOM, integrates with various DevOps tools in the ecosystem, and provides key information for each phase of the delivery process – Source, Build, Artifact, and Deploy. 

In the Source phase, for example, the solution could present all the vulnerabilities from the security scanning technology to the application owner. This would enable faster decision-making by the DevSecOps team while keeping all stakeholders in the loop to ensure no exceptions or bugs are introduced without management being informed. 

In the Build phase, the solution could connect with build or CI tools such as Jenkins or Travis CI to aggregate the data. If there is a policy violation, the DevSecOps team can quickly hinder the progress of a pipeline or inform the individual team to stall the release process. 

In the Artifact phase, the solution could empower the DevSecOps team with information about vulnerabilities related to dependencies by integrating with tools such as AquaSec, which provides information about supply chain security, malware protection, cloud security, etc.

Similarly, in the Deploy phase, the DevSecOps team must get security benchmarking information, such as from the Center for Internet Security (CIS) benchmarking. The system should fetch this data from tools that provide deployment verification and scores related to log analysis, metrics analysis, quality, reliability, and business impact. This information would help the DevSecOps team answer key questions before deciding to roll out software to an environment:

• Was the right image used in the deployment?
• What is the risk of the new application with regard to various dimensions?
• Who approved the deployment?

All this verification information from various tools is vital to securely progressing a release. 

At an operational level, a consolidated dashboard to track the DBOM has several benefits, including:

• Eliminate the need to fetch information from multiple sources, such as source code systems, CI systems, scanners, etc. 
• Independently monitor software delivery and control it from a security perspective. 
• See various security patterns from an organizational perspective (something that can be significantly different for developers working in silos) and implement policies to improve security posture.
• Bring various teams and stakeholders under the same umbrella using the DBOM dashboard to discuss and quickly resolve an issue.

4. Make SBOM a Part of Your DBOM for Dynamic Deployments

One of the core responsibilities of the DevSecOps team is maintaining a Software Bill of Materials (SBOM), which helps track associated security and license risks. SBOM includes a list of all the open-source and third-party components in a codebase, as well as:

• A list of names of the licensed application components.
• The versions of the ingredients used in the codebase.
• Their patch status

Today’s deployments are dynamic, with constantly changing infrastructure and dependencies. DevSecOps teams must automate the tracking and maintenance of SBOM in a single place. As delivery accelerates, teams must be able to react faster regarding each deployment and their respective vulnerabilities. By making SBOM part of DBOM, teams can see supply chain and security-related information in one place and formulate and implement policies faster.

5. Create Policies and Take Automated Actions in CI/CD

Once all the information is in place, the DevSecOps team should be able to create and implement policies, such as a policy to prevent a deployment based on a threshold of particular metrics related to code vulnerabilities or security scores. An automated mechanism to create new rules and enforce them in the delivery pipeline should be implemented to attain a risk-free software delivery process. Policy creation tools should easily integrate with CD tools like Spinnaker, Argo, and Jenkins. 

6. Central Exception Management for Distributed Teams

There may be instances within a specific environment when a scanner and risk-assessment tool misidentify a high risk to specific dependencies or libraries. The DevSecOps team should have a way to identify situations where open-source libraries are not exploitable given the setup and allow developers to proceed with their deployment activities. 

For example, developers testing applications with open-source libraries in a sandbox environment or Dev instance may introduce vulnerable open-source libraries. But if the environment is in a VPC behind a firewall, then the DevSecOps team may consider allowing those exceptions for a specific period. Having an exception management capability as part of the DevSecOps toolset is critical.

7. Audit and Attestation

DevSecOps conducts auditing exercises to ensure applications adhere to SDLC standards and regulations. Instead of manually fetching the information from ten disparate systems, they should be able to automate software auditing in their ecosystem via an audit and attestation dashboard that provides the who, what, and when of pipeline execution and policy violations. The solution should allow internal or external auditors to list, search, and filter on a date, deployment, environment, and event data collected from workflow tasks or pipeline activities and deployments.

Conclusion 

DevSecOps is a new concept with new responsibilities for enforcing security in CI/CD processes. Leading American organizations prioritize security and look to their DevSecOps team to lead the transformation. Due to the speed and scale of the CI/CD process, the DevSecOps team might find themselves in firefighting mode, ensuring security with manual checks of each release process. However, this manual approach at each application level can be overwhelming, leading to the risk of missing key security checks from an organizational perspective. To ensure security and avoid the burnout that comes from manual activities, DevSecOps teams must understand the above best practices and adopt the new tools and technologies necessary to implement them.