A MAP for Kubernetes Supply Chain Security

The sharp increase in software supply chain attacks has made securing the build and delivery of software a critical topic. But what does this mean for Kubernetes DevOps teams tasked with securing their continuous delivery pipelines and clusters? To get started with securing a Kubernetes supply chain there are four things you will need to consider: Artifacts, Metadata, Attestations, and Policies (A-MAP). Let’s dive in!

In a prior post, Dan Lorenc (Founder/CEO at Chainguard) and I discussed container image signing and verification. 

While signing is a good first step in securing software supply chains, it is not enough for higher levels of security required by emerging standards such as Supply Chain Levels for Software Artifacts (SLSA). Signing does not tell us how and where the software was built or what it is made of.

Software build systems produce artifacts and metadata. Verifying build integrity and software component security attributes requires attestations and policies.  Each of these items plays an important role in a secure software supply chain.

Artifacts

Software builds produce artifacts for installation and execution. The type and format of artifacts vary depending on the software. They can be packages, WAR files, container images, or other formats. 

Metadata

Metadata is data that describes the software artifacts. There are three types of metadata that should be considered for software supply chain security:

Provenance Data

Provenance means place of origin. For software systems, provenance data is information about the build system that produced it. This includes machine identities, build software and process information, CI/CD tool information, and any other details useful in verifying the build environment. 

While no standards exist for provenance data, in-toto, a CNCF project that provides a set of tools and specifications for software supply chain security, allows custom predicate types which can be used to represent provenance data. We will discuss in-toto attestations in more detail in a later section.

Software Bill of Materials (SBOMs)

An SBOM is the list of ingredients (i.e., other software packages and libraries) that the software contains. There are two leading formats for SBOMs: CycloneDX and SPDX.  

The National Institute of Telecommunications and Information Administration (NTIA) has published a guide, Survey of Existing SBOM Formats and Standards, comparing these standards as well as Software Identification Tags (SWID) which while not a full SBOM offers a standard way to identify software components.

Vulnerability Scan Reports

Vulnerability scanning identifies known security issues and is a primary step in securing software systems. Scans should be performed as early as possible in the software delivery lifecycle. Software builds should fail when new or unaddressed vulnerabilities are detected. 

However, not all vulnerabilities can or must be fixed and hence someone knowledgeable with the software must review vulnerabilities and approve allowed vulnerabilities. The Vulnerability Exploitability Exchange (VEX) format is often used with SBOMs to provide an assessment of how known vulnerabilities impact a software component. 

While most other metadata is immutable, vulnerability scan reports should be refreshed periodically as new vulnerabilities can be reported after a software system has shipped.

In addition to provenance data, SBOMs, and vulnerability scan reports, other types of metadata such as SAST and code review reports can also be created for organizational and regulatory compliance.

Attestations

Metadata provides useful information, but how can the metadata itself be trusted? In a real-life audit, an independent auditor reviews information and produces a report to attest to the accuracy and reliability of the information. Similarly, in a software supply chain, an attestation is signed metadata that is produced by a trusted person or system. In a secure software supply chain, the build system signs metadata like provenance data, vulnerability scan reports, and SBOMs to produce attestations. 

Code and image signing can be thought of as a type of basic attestation. Signing a software package or container image simply means some trusted entity attests to its integrity. However, signing does not provide any additional guarantees to consumers. Hence, it’s more meaningful to sign metadata that can be used to inspect and verify information about the software. The attestations can be used by consumers to establish trust in the software system.

The in-toto project has defined a standard format for attestations, which while still in development is being rapidly adopted as a way to represent authenticated metadata. 

Tools like sigstore Cosign can sign in-toto predicates and attach them to a container image as attestations. Policy engines like Kyverno can be used to verify these attestations.

Policies

Artifacts, metadata, and attestations are produced by the build system, but how should consumers use this information and enforce software supply chain security for their organizations? 

Policies are the answer to that question, and the final step in a secure software supply chain. Policies help automate and address the challenge of verifying build information such as attestations. 

For example, an image verification policy can declare what an organization requires for any image that is deployed into its production environment. Here is an example:

1. All images must contain the following attestations in the in-toto attestation format:
   • Provenance data
   • Vulnerability scan reports
   • SBOMs
2. Vulnerability scan reports should be in VEX format.
3. Vulnerability scan reports should be updated daily.
4. No high severity vulnerabilities are allowed.
5. SBOMs should be in CycloneDX or SPDX format.

Policies should be enforced prior to deployment, and periodically via runtime scanning. 

Cloud-native systems like Kubernetes are extensible and offer the concept of admission controls to allow verification of components before they are deployed in a cluster. Kubernetes policy engines like Kyverno (https://kyverno.io) offer flexible policy formats to verify images and configurations prior to deployment and at runtime via continuous scanning.

Conclusion

In this post, I introduced the four items required for Kubernetes DevOps teams to understand software supply chain security:

• Artifacts: Build systems produce various artifacts that install or execute software.
• Metadata: Metadata is used to describe software and the build environment. Provenance (origin) data, SBOMs, and vulnerability scan reports are the essential set of metadata required to assess security risks for software.
• Attestations:  Authenticated metadata is used to attest to the integrity of a software system. Both custom and standardized metadata can be converted into attestations.
• Policies: Policies check and enforce organization standards. Policies should be automatically enforced prior to deployment and via runtime scanning.

Modern build systems and CI/CD systems offer customizable workflows that can integrate tools to produce standards-based attestations. Cloud-native systems like Kubernetes and its ecosystem of projects and tools provide policy engines to verify attestations and enforce software supply chain security. Signing artifacts is a type of attestation and a good first step for securing software supply chains. However, signing by itself does not provide adequate information to verify and enforce security levels for software components. 

Standardized attestations using the in-toto attestation format allow CI/CD system to produce metadata, tools like Cosign to create attestations, and policy engines like Kyverno to verify the attestations at runtime.

As a follow-up post, I will provide a complete example of how to implement a secure software supply chain for Kubernetes that implements these concepts. 

Get Involved

With over 2000 GitHub stars and 150M downloads, Kyverno is a CNCF project and the policy engine designed for Kubernetes. With Kyverno, policies are managed as Kubernetes resources, and no new language is required. This allows using familiar tools such as kubectl, git, and kustomize to manage policies. Kyverno policies can validate, mutate, and generate Kubernetes resources plus ensure OCI image supply chain security with integrations for Sigstore Cosign and in-toto attestations. If you would like to learn more about Kyverno, follow our GitHub repository to stay updated.