From Vulnerability to Victory: Defending Your CI/CD Pipeline

Automated CI/CD (continuous integration/continuous delivery) pipelines are used to speed up development. It is awesome to have triggers or scheduling that take your code, merge it, build it, test it, and ship it automatically. However, having been built for speed and ease of use means that most pipelines are not inherently built with security in mind. Since the pipelines usually need to have access to the internet to download dependencies, and to your various secrets to upload to your production environment, it means that once such a pipeline is compromised, the attacker has a wide range of options to disrupt your operation or exfiltrate information or secrets.

All of the stories presented in this article describe breaches in prominent CI/CD tools. The fact that most companies rely on such tools means that, like many other software supply chain attacks, all the bad actors need is to breach a single target to get a vast blast radius.

Let’s take a look at a few prominent stories from the past few years demonstrating the vulnerabilities inherent in this attack vector. At the end of the article, I’ll make sure to offer my recommendations to harden and protect your pipelines, no matter what tools you’re using.

The CircleCI Breach

In January 2023, CircleCI, a continuous integration and delivery platform, disclosed a security breach that allowed an attacker to gain unauthorized access to the company’s infrastructure. The attack was initiated by a phishing email that tricked an employee into sharing their credentials, which the attacker then used to access the company’s systems. The employee was utilizing 2FA, but the attacker was able to grab a session cookie once the employee had already logged into the system, allowing them to impersonate the employee and eventually escalate access to a subset of CircleCI’s production systems.

The attacker accessed some customer data, including names, email addresses, and billing information. The company reported that no code or intellectual property was accessed, and no customers reported any unauthorized access to their accounts or data.

CircleCI responded quickly to the breach, conducting an investigation and working with law enforcement and cybersecurity experts to identify the scope of the attack and remediate the vulnerabilities that allowed it to occur. The company also reset all employee and customer credentials and implemented additional security measures, such as increased monitoring of its systems.

DevOps Disrupted: The Argo CD Security Breach

Argo CD is a popular open-source tool for the continuous deployment of applications to Kubernetes clusters. In February 2022, a new vulnerability was discovered in Argo CD that allowed unauthenticated users to access sensitive information about Kubernetes applications managed by the tool. The vulnerability was caused by a misconfiguration in Argo CD’s API server that allowed unauthorized users to execute requests to the API and retrieve information such as secrets, configurations, and application metadata. As long as the attacker had permission to create or update applications and knew or could guess the full path to a file containing a valid YAML they could create a malicious Helm chart and keep consuming YAML files as values until they found a juicy piece of data that would normally be inaccessible.

The vulnerability was assigned a CVSS score of 7.5 (High severity) and affected all versions of Argo CD up to and including version 2.1.4. Argo CD released a patch for the vulnerability in version 2.1.5, and users were advised to upgrade to this version as soon as possible.

The vulnerability was deemed particularly concerning because Argo CD is often used to manage critical applications and infrastructure in production environments. The leak of sensitive information could have allowed an attacker to gain access to sensitive data or take other malicious actions that could impact the availability or security of the applications.

Uncovering the Codecove Breach: Lessons Learned

Codecov is a code coverage tracking and analysis tool used inside CI/CD pipelines that allow developers to measure and analyze the effectiveness of their tests. As published in Codecov’s security update: 

“On Thursday, April 1, 2021, we learned that someone had gained unauthorized access to our Bash Uploader script and modified it without our permission. The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script.”

The Bash Uploader is used by customers to upload their code coverage reports to the Codecove platform. With this access, the attacker modified the Bash Uploader script by adding malicious code that collected environment variables, authentication tokens, and other sensitive data from the user’s system during the upload process. This data was then sent to a remote server controlled by the attacker.

According to Codecove, the breach impacted approximately 1% of its customer base, which included major companies in the technology, finance, and healthcare industries. The company confirmed that no customer code or code coverage reports were altered during the breach, but that user authentication tokens and other sensitive information may have been compromised.

Codecove took immediate action to remove the malicious code and alert affected customers to reset their authentication tokens and take other security measures. The company also launched an investigation into the incident and engaged with law enforcement and cybersecurity experts to identify the source of the attack.

How Can You Defend Your Pipeline?

As mentioned above, CI/CD pipelines are built for speed and automation, not necessarily for security. The fact that three big and respected companies have all fallen victim to some sort of attack, potentially exposing their clients’ information, demonstrates the vulnerability of your own pipeline and data.
No matter what tools or CI/CD platform you’re employing, you can do a few things to improve your security and reduce the possible damage in case a malicious actor does manage to gain access to your pipeline or network.

• Threat modeling: Threat modeling is a structured approach to identifying potential security threats to a system or application and designing appropriate countermeasures to mitigate those threats. As an exercise, imagine that someone gained access to your pipeline. What environments can they now access? What secrets and keys can they see and use? Can they modify your code? Influence tests? Pull files from the web or run a reverse shell? Even if you think that you have sanitized your pipeline and properly segmented network access, it doesn’t hurt to imagine and then check just so you’re aware of a worst-case scenario. You might be surprised what secrets and access options are hidden in plain sight inside your pipeline platform or tooling.
• Network segmentation: Network segmentation is the practice of dividing a larger network into smaller, more secure subnetworks or segments, each with its own security controls and policies. The purpose of network segmentation is to increase the security of the overall network by limiting the scope of a potential security breach and minimizing the potential impact of an attack. Breaking a network into smaller segments limits the lateral movement of attackers, and reduces the risk of unauthorized access or data leakage.
  With the increasing popularity of phishing attacks, any of your developers or other employees is may fall victim to such a scam — we’re all human. Assuming your developers’ credentials could be used by malicious actors means you need to make sure that the overwhelming majority of developers shouldn’t have the kind of access that would allow them to single-handedly exfiltrate secrets, implant malicious code in a production image, or just push their own version of a production code uncontested. Making sure each individual has the least amount of access needed for their work is time-consuming, and the temptation to just give everyone admin access and be done with it is strong. Don’t be tempted by the dark side — follow the rules of zero trust and minimum privileges.
• Monitoring and alerting: Following from the last point, even if you have thoroughly drilled your developers to be weary of phishing and other social engineering scams, a breach could still happen. You don’t know when or how but you should be prepared to find out about it at the very least. Since most pipeline environments are ephemeral, that means that once the job is done, there is no trace of evidence to be found on what happened unless you create such traces yourself.
  Make sure to log everything happening in your pipelines, on each PR, merge, build, and test done. Make sure the users’ information is logged as well, to be reviewed in case a problem requires such a check. Any changes to configuration files or the environment itself should also be logged. The goal here is to be able to do a clear post-mortem on a breach to be able to tell what happened and how. Determine in advance which events should warrant an alert and ensure the right people get that alert. Be wary not to flood people with useless or false alerts — that would cause alert fatigue that would just lead to them ignoring the alerts or responding much later than advisable. Obviously, none of these logs should contain any open secrets or keys — which leads to the next bullet point:
• Secrets management: Make sure you’re using a secrets management tool of some sort. At the very least it would make it easier to rotate your secrets and passwords if a breach does occur. It should also do the work of redacting open secrets and access keys found in the pipeline from any logging done on the system. At no point should anyone unauthorized have access to such information and they should definitely not be able to change it.
  Secrets management is becoming increasingly important as organizations increasingly rely on cloud-based services, containerized applications, and other distributed environments that require the secure sharing and management of secrets across different systems and applications.
• Use the RBAC principle combined with least privilege: The principle of Role-Based Access Control (RBAC) is to provide access to system resources based on a user’s assigned role or job function within an organization. In RBAC, users are assigned roles that define the permissions and access rights they have to various system resources, such as files, folders, or applications. The principle of Least Privilege, on the other hand, is the practice of granting users the minimum level of access and privileges necessary to perform their job functions. This means that users only have access to the resources they need to perform their specific tasks, and no more. RBAC and Least Privilege are often used together as complementary security principles. In RBAC, users are assigned roles that have the appropriate level of access to the resources they need to perform their job functions, and the principle of Least Privilege ensures that users only have access to the minimum level of resources necessary to perform their specific tasks. Together, these principles help to maintain a secure and well-managed system with minimal risk of unauthorized access or data breaches. As an added bit of security, you can set it so that critical actions in the system require multiple user authorizations to go through. This approach should be taken with care since it has the potential to significantly slow down development work. However, for critical updates, such as deleting a main branch or modifying a dependency list, you should make it so that at least two people with the appropriate access need to approve it.

Looking Forward

No one is going to stop using CI/CD and other automation tools to speed up their work. We live in a world where we constantly strive for faster and faster code updates. We just need to make sure we do it in a security-aware way and are not compromising our code and production environment along the way.

One of the most important things you can do is just to give some thought to what could happen if unauthorized people gain access. Once you become aware of the danger and the different places in your pipelines and network that are vulnerable, I trust you’ll be able to take the right steps to plug the potential leaks.