Log4Shell: A Case for Trusting Open Source – With Guardrails

Along with a host of frenzied updates and patches, Log4Shell brought something else to the table: an intense renewed scrutiny among business leaders and governments around “open source.” What most of these critics are not aware of is that much of the software powering their success isn’t created by commercial vendors, but is created by volunteers and that some of their most critical systems use open-source software. Furthermore, most critics can’t confidently point to a list of all the open-source software powering their own success. 

Similar to the response we’ve seen to major incidents like HeartBleed, Dirty Cow, and the Equifax experience with Apache Struts, governmental reviews are underway, and some are seeking to replace the “bad open-source component” – in this case, log4j – with a “more secure alternative.” But there is an important aspect of open source in modern society that is being overlooked in these scenarios – it’s highly trusted.

In fact, this inherent trust in open source is so strong that in some organizations, software teams download and use it as-is, without significant security reviews. This is not uncommon – rarely do businesses perform the same level of security review on software downloaded from the internet as they do for software they create or that they procure from a commercial vendor. 

For those doubting this statement, ask yourself: who in your organization reviewed the code for Kubernetes? Who reviewed libxml? Did anyone review the open-source database your applications depend on for security issues? Even if such reviews did occur, there’s a strong likelihood that this is not a reoccurring review with each new update. And, that’s also a reflection of the trust placed in those open source efforts.

The Appeal (and Risk) Behind Adopting Open Source Software

Although there are varying reasons organizations adopt open source technologies, there are three primary ones. First, open-source software often comes without a licensing fee. This means the individual adopting the software doesn’t have to handle nuisances like budget approvals or employer procurement reviews. Second, the chosen open source technology was likely developed by an expert, which eliminates the oftentimes complex and expensive process of hiring an expert for the in-house team. Lastly, the chosen technology often represents a popular solution to a problem. Kubernetes is one example: while there are many options available for container orchestration, Kubernetes is the most popular — meaning that its implementation is trusted for use by both peers and competitors.

Often missing from the adoption discussion is that open-source software, similar to commercial software, is made up of multiple components. Linux is not a single executable. Nor is there a single Kubernetes executable. Each has a significant number of dependencies, sub-systems, and support services. This also holds true for mobile applications, IoT firmware, and even simple business logic functions deployed on AWS Lambda — they all have dependencies that are required for the application to function properly. When people talk about “software supply chain”, it is this list of dependencies that people are referring to, and not an arbitrary directory listing of files making up that application. And it is this list of dependencies where the greatest risk from open source usage originates.

The Variable Nature of Open Source Code

Open source projects popular enough to become a de facto standard, like Kubernetes, have a large number of volunteer developers working on the code. Some of these developers are even employed by companies using Kubernetes to singularly work on Kubernetes, which translates into resiliency for Kubernetes and the Kubernetes ecosystem. Should one developer quit or retire from the project, the project moves on. The departure of even key team members on popular projects is something that can be handled with minimum disruption. 

The same can’t be said for smaller projects. GitHub has millions of projects for which the number of developers is in the single digits. And GitHub is far from the only repository of open source activity. When the departure of a single developer for those projects happens, it often means losing someone who understands exactly why specific areas of code were written the way they are. It is these projects that are most commonly found on the list of dependencies for an application, and they often perform fundamental actions like writing log data—as is the case with log4j. 

Given the variability in the number of developers for open source projects, it’s not surprising to see variability in how individual projects respond to a security incident. Some have very well-defined security processes that are a reflection of their prominence, like Kubernetes, Android, or the Xen Project. A well-defined security process likely also means that when there is a patch, it will be associated with a security disclosure in CVE format. But others will treat security issues as another bug to be resolved at an indeterminate point in the future. Projects that treat security issues like bugs are far more likely to simply fix the bug and not issue a disclosure. This variability makes it difficult for businesses to define the risk in their software supply chains and develop reasonable policies to mitigate that risk.

The Path to Making Open Source Security a Priority

The path to mitigating the business risk associated with using open-source software—which is in reality software risk—starts with accepting that the business benefits from the use of open-source software. In reality, the main business risk comes down to the fact that open-source software is managed differently than commercial software. Procurement and patch processes are different between commercial and open-source software, so organizations should not expect them to be managed similarly. 

The de-risking effort then needs to start with a comprehensive asset inventory of all software used by the business, regardless of its origin or the procurement process used during acquisition. That inventory makes it achievable to determine which open source components are used by each asset. That’s done using what’s known as Software Composition Analysis (SCA) tools. Since the software can be both compiled into a binary and available in source form, it’s critical that the chosen SCA tool can process both binary and source-based assets with equal capabilities. Since that asset inventory likely has thousands of entries, organizations will want the SCA tool to proactively alert you to any changes in security risk within your assets without requiring a rescan of the assets. 

It’s at that point when you can define your run book for how to address the changes in risk from a new security disclosure. At the end of the day, it’s hard to patch something you didn’t know you had—and you never know when a new piece of software was created with a vulnerable component.