The Dark Side of DevSecOps and Why We Need Governance Engineering

For today’s software organizations security has never been more top of mind. On one side, there is the present and growing threat of being hacked by malicious actors, set out in Crowdstrike’s recent Global threat report. And on the other, there is a wave of cybersecurity regulation from the government to mitigate such cybersecurity vulnerabilities. Software organizations feel the heat from both sides as they work to improve their security posture in ways that will also achieve audit and compliance with new rules. 

To meet these challenges, we hear a lot about the software supply chain and the importance of securing it with DevSecOps approaches and golden paths to production. Since the Biden administration's executive order in 2021, we’ve heard a lot about software bills of materials (SBOMs), and new products, services, and vendors have sprung up to help us with them. But do DevSecOps and supply chain approaches meet our cybersecurity challenges head-on?

We Need To Talk About Dark Deploys

Responsible organizations already spend a lot of time and resources building secure pipelines, implementing best practices, and testing applications for vulnerabilities. Despite all these efforts, there's a hidden risk that often goes overlooked in the supply chain story: off-pipeline changes. 

What happens when software changes aren't properly vetted and approved through the established security process? What if they introduce vulnerabilities that go undetected until it's too late?  What if a nefarious insider or attacker runs workloads in your infra? 

I want to emphasize the importance of tracking and monitoring all deployments, especially those bypassing standard security protocols. So let’s discuss some of the risks and consequences of unapproved deployments and explore some best practices for keeping your system secure and compliant, no matter how fast you need to move.

How Secure Are Golden Pipelines/Golden Paths?

In regulated industries, governance in development is a key focus when it comes to securing the software delivery pipeline. Teams are often required to follow strict guidelines to ensure the security and integrity of their systems.

Many organizations do this with what’s often referred to as a “golden pipeline” that incorporates all of the necessary security checks like pull request checks, SAST, DAST, software composition analysis, and so on. A typical example of a golden pipeline looks something like this:

Most teams follow these golden pipelines to production to ensure that their software delivery process is secure and compliant. And there are a lot of benefits to following this approach:

• It takes the burden off each team to figure out how to comply.
• It allows all the stakeholders to collaborate and reach a consensus once-and-for-all.
• It makes audit and change management easier.

However, while DevSecOps processes like these have been effective in controlling and monitoring the delivery pipeline, they do not secure the entire software development lifecycle. Bad actors can bypass these controls and introduce unapproved changes directly into production, putting the entire organization at risk. So let’s probe a bit further and find out where the vulnerabilities are. 

The Limitations of DevSecOps Pipelines

DevSecOps pipelines and golden paths are put in place to ensure that changes made to a system follow a defined process and are authorized before deployment. This helps maintain system stability, ensure compliance, and mitigate risks.

But pipeline controls have one obvious limitation when it comes to ensuring the security and compliance of an entire software system. They can only ensure security and compliance for changes that have gone through the pipeline. They do not account for bad actors who access production by going around the golden path. 

There are several key security questions that cannot be answered in the pipeline:

• How do we discover workloads that haven’t gone through our pipeline?
• What happens if an internal developer has the keys to production?
• What happens if we are breached?
• What happens if our deployment process has silent failures?

Think of a golden pipeline as a river running into a lake. Monitoring what’s flowing in the river does not guarantee the quality of the water in the lake. You need to monitor the quality of the water in the lake too!

Closing the Security Loop With Runtime Monitoring

These security questions that can’t be answered in the pipeline can be answered in runtime. Monitoring production is the only way to be sure about what’s actually running in your environments and adds an extra layer of security on golden pipelines. It’s also quite straightforward:

• Record every approved deployment in the pipeline.
• Monitor (and record) what is running in production.
• Raise the alarm if any unexpected workloads are running.

By monitoring and recording the state of production, organizations can detect any unauthorized changes that have bypassed the change management process. This closes the loop on security, ensures continuous compliance in your environments, and greatly mitigates threats and risks. 

The Benefits of Security Audits (and Auditors)

For many teams, particularly in regulated industries like finance and healthcare, software delivery must be managed according to a specific process to manage risks. And to ensure that the process is followed for each change, a change management process is added for verification before deployment. Traditionally, it is a gated process of manual controls and approvals. 

Change management implemented as a gate has a lot of negatives:

• Increases lead time for changes.
• Leads to larger and more risky releases.
• Requires manual audits to ensure compliance.

But taking a golden pipeline approach to the change management process builds compliance with every change. By recording the evidence for every test, security scan, pull request, etc., as they happen in the pipeline, proof that the delivery process is being followed and risk controls are being met can be gathered automatically. As a result, it’s more reliable, faster and removes the need for a manual gate. 

This approach turns subjective processes into objective, high-efficacy risk management.  It also has the added benefit of enabling automated, continuous auditing. You don’t have to spend time gathering the paperwork for an audit trail when all of the evidence has already been captured in your pipelines and then securely stored. 

Full Chain of Custody With Governance Engineering

Governance engineering is an alternative approach to traditional change management processes. It brings compliance, risk, security, and development teams together to define standards and automate controls, evidence gathering, and approval work. It leaves humans in the loop for governance while providing audit-ready, continuous delivery compliance.

With governance engineering, organizations can identify and address non-compliant changes immediately instead of waiting for audit time. It’s faster, more effective, and provides better risk mitigation than traditional change management processes.

Conclusion

DevSecOps approach effectivcontrolstrol security in the delivery pipeline, but they fall short of securing the entire software development lifecycle. Monitoring production is a powerful addition that will help you to properly secure a  DevSecOps approach. By keeping a close eye on production, organizations can detect any unauthorized changes and address them immediately. 

With Runtime monitoring and governance, engineering organizations can bring compliance, risk, security, and development teams together. In this way, the whole tech organization can work to define standards and automate controls, evidence gathering, and approval work to ensure a much more secure DevOps environment.