Protect the Pipe! Secure CI/CD Pipelines With a Policy-Based Approach

In the last few years, there has been a sharp rise in software supply chain attacks, where malicious actors target software build and deployment environments to infiltrate production systems. There are three reasons that have contributed to this increase:  

1. Improved security for production systems has forced attackers to look for other avenues. The improvements may be due to the increase in cloud and managed services and general security awareness and availability of tools.  
2. With the adoption of programmable infrastructure and Infrastructure-as-Code (IaC), build, and delivery systems now have access to production systems. This means a compromise in the build system can be used to access production systems and, in the case of a software vendor, access to customer environments.
3. Applications are increasingly composed of hundreds of OSS and commercial components. This increases the application exposure and presents several ways to add malicious code to an application.

All of these factors contributed to attackers shifting focus to Continuous Integration and Continuous Delivery (CI/CD) systems as an easier target to infiltrate multiple production systems. Therefore, it is essential that organizations give equal consideration to securing our CI/CD pipelines, just as they do their production workloads. 

Similar to our business applications, CI/CD pipelines also have their own life cycle, including stages like pipeline composition, configuration, invocation, execution, and completion. For comprehensive coverage, security controls need to be embedded and exercised across all these defined stages of the pipeline. 

There are a number of frameworks available for implementing CI/CD pipelines, including Tekton, GitHub Actions, Jenkins, ArgoCD, and many more. In this post, we will scope our discussion around Tekton, which is one of the most popular emerging cloud-native solutions for CI/CD pipelines. However, the primary focus is on illustrating various security features of Kyverno–a Kubernetes native policy engine–and how they can be applied effectively for improving the overall security posture of our CI/CD pipelines.

Tekton, Kubernetes-Native CI/CD

Tekton is a powerful yet flexible Kubernetes-native open-source framework for creating CI/CD systems. It lets you build, test, and deploy across multiple cloud providers or on-premises systems by abstracting away the underlying implementation details. 

The Tekton Pipelines project allows us to define a complete CI/CD pipeline in a declarative style as Kubernetes-native resources. The following are the primary resources required to construct a pipeline:

• Pipeline: creates an overall orchestration of the pipeline that primarily includes a list of allowed `tasks` and their order of execution. A Pipeline also declares shared resources (secrets, workspace, configmaps, etc.) across tasks.
• Task: embodies one or more execution actions (or `steps`), resources used for executing those actions (e.g., container images, environment properties, etc.), and expected result paths.

Typically, these resource definitions are managed through independent manifest files (typically maintained as YAML files) and are even shared through open-source catalogs like the Tekton Hub. 

There is also a new packaging and distribution option available in the form of a `bundle` that lets us store one or more of these resource manifests in OCI format. A Tekton Bundle brings consistency in packaging our pipelines and our production application in a standard OCI format. This also allows us to implement a common security framework, including applying policies like Kyverno’s image verification rules to OCI bundles and validation checks to Tekton resources.

There is another initiative called Tekton Chains that automates the monitoring of task executions and provides attested provenance records for pipeline operations and generated artifacts like container images. These attestations are then used for validating the artifacts at runtime.  

Kyverno, Kubernetes-Native Policy Management

Kyverno is a policy engine designed for Kubernetes. Kyverno can validate Kubernetes configurations and block API requests that create insecure or non-compliant configurations. Besides validation, Kyverno can also mutate and generate configurations for new or existing resources. This set of features makes Kyverno a powerful tool for automating security concerns in Kubernetes. And best of all, Kyverno uses Kubernetes resources for policies and policy results and does not require learning a new language. 

A Threat Model for Tekton

Kelly Shortridge wrote about security decision trees as a way of organizing security threats and prioritizing mitigations. Deciduous is a web application to help build attack trees. 

The following is a basic attack tree for Tekton:

Here is how to read the attack tree:

• The gray boxes represent facts. For example, “Pipeline executes Tasks” simply states the seemingly obvious that in Tekton, a Pipeline resource executes one or more tasks.
• The pink boxes represent threats. “Attacker runs an unknown Pipeline” represents a potential threat introduced by the prior fact.
• The blue boxes represent mitigations. “Require signed Pipeline bundles” provides mitigation to the previous threat.

Kyverno Policies to Mitigate Threats

Based on the threat model, we identified the following policy checks for Tekton pipelines:

1. Block direct execution of Task (i.e., users cannot directly create a TaskRun resource).
2. Require Namespaces for  PipelineRun and TaskRun resources because in Kubernetes, Namespaces enable isolation and segmentation.
3. Generate defaults for new Namespaces. This includes Secrets, Roles, a default NetworkPolicy that limits egress and ingress traffic, and a PersistentVolumeClaim that generates a shared volume for tasks executed within the Namespace.
4. Require that pipelines and tasks are part of a Tekton bundle that is signed using a trusted authority. Tasks and pipelines that are not from a signed bundle will be blocked.
5. Check all Task images for vulnerabilities by requiring an attestation with a scan report and no vulnerabilities that are Critical or High in severity.
6. Require that each Task step has a security context defined. This prevents the execution of privileged containers, or containers with root privileges, to mitigate container escapes.

Each of the above checks and actions is implemented by a Kyverno policy.

As an example, here is the Kyverno policy that requires pipelines to be executed from a bundle:

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-bundle
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: check-pipeline-run
    match:
      resources:
        kinds:
        - PipelineRun
    validate:
      message: "a bundle is required"
      pattern:
        spec:
          pipelineRef:
            bundle: "?*"

This Kyverno policy requires that all Tekton bundles must be signed using a trusted authority:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: signed-pipeline-bundle
spec:
  validationFailureAction: enforce
  webhookTimeoutSeconds: 30
  rules:
  - name: check-signature
    match:
      resources:
        kinds:
        - PipelineRun
    imageExtractors:
      PipelineRun:
        - name: "pipelineruns"
          path: /spec/pipelineRef
          value: "bundle"
          key: "name"
    verifyImages:
    - imageReferences:
      - "*"
      attestors:
      - entries:
        - keys: 
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEahmSvGFmxMJABilV1usgsw6ImcQ/
              gDaxw57Sq+uNGHW8Q3zUSx46PuRqdTI+4qE3Ng2oFZgLMpFN/qMrP0MQQg==
              -----END PUBLIC KEY-----

The complete set of policies is available in this Git repository. 

Conclusion

As supply chain attacks become more sophisticated, it is important to implement security best practices for building and deployment tools. Tekton provides a powerful framework for CI/CD, and with extensions like Tekton Chains, it is now possible to secure build artifacts. 

However, securing Tekton itself is paramount. Since Tekton is based on Kubernetes, Kyverno becomes the ideal policy engine to secure and automate the mitigation of key threats. 

Using Kyverno, we can manage policies to require Namespace-based isolation and generate security resources for Tekton pipelines. Kyverno can also enforce best practices and require signed OCI artifacts like Tekton bundles. 

This powerful combination of Tekton and Kyverno enables a new level of security and automation for the software build and delivery systems.