Agent 008: Chaining Vulnerabilities to Compromise GoCD

GoCD is a popular Java CI/CD solution with a large range of users, from NGOs to Fortune 500 companies, with billions of dollars in revenue. Naturally, this makes it a critical piece of infrastructure and an extremely attractive target for attackers. In a previous article, Agent 007: Pre-Auth Takeover of Build Pipelines in GoCD, the SonarSource R&D team demonstrated how unauthenticated attackers could impersonate build agents and access features that were previously protected by authentication mechanisms (CVE-2021-43287), leading to the disclosure of credentials and sensitive tokens for third-party services. 

In this follow-up article, I describe three additional vulnerabilities discovered and responsibly disclosed by the SonarSource R&D team in GoCD 21.2.0 and below. First, a vulnerability that can be used by attackers impersonating build agents to force administrators to perform security-sensitive actions without their knowledge (CVE-2021-43288). Then, two additional vulnerabilities could be chained, with the first one fully compromising the targeted instance by executing arbitrary commands (CVE-2021-43286, CVE-2021-43289) on the server hosting GoCD. These findings are already addressed by the latest release of GoCD: this article aims to share the root cause analysis and insights on how they could be exploited. 

A threat actor taking advantage of these vulnerabilities could gain control of components within a release pipeline and leak intellectual property or include backdoors in the company's software. As an example, think about the SolarWinds hack, where attackers gained access to the software delivery pipeline and added a backdoor to critical software, leading to one of the most impactful supply-chain attacks thus far.

Impact

These three additional vulnerabilities in GoCD can be exploited by attackers who bypassed the mandatory authentication and obtained Agent privileges as presented in Agent 007: Pre-Auth Takeover of Build Pipelines in GoCD using CVE-2021-43287. 

The first one is a Stored Cross-Site Scripting vulnerability that allows attackers to impersonate administrators after they visit a poisoned job status page. To replicate what real-world attackers could do, I identified two post-authentication vulnerabilities that can lead to the execution of arbitrary commands on the server when chained with the cross-site scripting vulnerability. Here is a representation of how they could be connected by attackers to compromise the server:

Attackers exploiting these findings could leak API keys to external services such as Docker Hub and GitHub, steal private source code, get access to production environments, and overwrite files that are being produced as part of the build processes, leading to supply-chain attacks.

All my findings, including the ones presented in the first article, were addressed in GoCD v21.3.0, available since October 26th. 

Technical Details

The three findings I describe in this article are all related to agent tasks and the way they communicate their results back to the GoCD server. From an architectural perspective, agents can be considered a special kind of user with a different HTTP API and means of authentication. They are identified with a UUID transmitted in the X-Agent-GUID header and an HMAC of this value in Authorization.

They get new jobs by calling /go/remoting/api/agent/get_work at regular intervals with a GetWorkRequest packet. When a pipeline should run, and an agent is chosen for the workload, the server provides the agent with all the necessary information. This includes the commands to run and the secrets and environment variables to use. 

While performing the pre-defined actions for their tasks, they send their status (e.g., building, passed, etc.), the console output, and eventual files and folders resulting from the build (also named “artifacts”) back to the server. These two last elements are sent over the /go/remoting/files/ endpoint.

CVE-2021-43288 — Cross-Site Scripting on Job Status Page

This first finding is related to the job status page, which displays everything about jobs, including tests, a tree display of artifacts (files, folders), and a console-like presentation of logs.

Let’s take a look at the source code behind this feature. GoCD implements its own server-side presentation layer: controller code has to create and fill HtmlElement objects, which will later be sent back to the client after being processed by a HtmlRenderer.

The rendering of the Artifacts tab is implemented in DirectoryEntries.java. At [1], it iterates over DirectoryEntry objects and call their toHtml() method and passes it to the presentation renderer at [2]:

common/src/main/java/com/thoughtworks/go/domain/DirectoryEntries.java

public class DirectoryEntries extends ArrayList<DirectoryEntry> implements HtmlRenderable, JsonAware {    
   @Override
   public void render(HtmlRenderer renderer) {        
     // [...]
     for (DirectoryEntry entry : this) {   // [1]
     	entry.toHtml().render(renderer);  // [2]
     }    
}

For both directories and files in the artifacts list, the final HTML code is generated based on the entry name without further sanitization: 

common/src/main/java/com/thoughtworks/go/domain/FileDirectoryEntry.java

public class FileDirectoryEntry extends DirectoryEntry {    
    @Override
    protected HtmlRenderable htmlBody() {
        return HtmlElement.li() 
        // [...]                
        .content(getFileName())

Attackers impersonating agents can exploit this weakness by sending artifacts with malicious names to inject arbitrary HTML elements into the page, such as <img%20src=x%20onerror=alert(document.domain)>. 

As shown in the capture below, the persistently stored payload will then be executed as soon as the job status page is opened by the victim. This page is likely to be visited by administrators if attackers deliberately fail CI jobs to get their attention.

Through this vulnerability, attackers are able to execute arbitrary JavaScript code in the victim’s browser, including initiating further HTTP requests with the victim's privileges. 

The maintainers addressed this vulnerability in f5c1d2a, in which they introduced the use of org.apache.commons.text.StringEscapeUtils to escape the names of files and folders during their rendering as HTML elements.

Executing Arbitrary Commands on the Server

With the help of the Stored Cross-Site Scripting vulnerability I described in the first section, attackers could force authenticated users to perform arbitrary actions without their knowledge, like disabling authentication or exploiting vulnerabilities that would not be reachable by the attacker otherwise. 

To demonstrate this risk, the SonarSource Vulnerability Research team identified two additional vulnerabilities that can be chained with the Stored Cross-Site Scripting in order to gain arbitrary code execution on the GoCD instance.

The first finding is related to the way artifacts are written on the local filesystem: a parameter used by the application to craft the final destination path of the artifact is not validated. This behavior allows attackers to write files with arbitrary content to an arbitrary location.

A second vulnerability was discovered in the way GoCD processes the URLs of remote code repositories. Because of insufficient validation of these values,  the behavior of external commands invoked by GoCD can be altered.

CVE-2021-43289, CVE-2021-43290 — Path Traversal in Artifacts Upload

Vulnerable Code

Console output and artifacts are sent over the /go/remoting/files/ endpoint. The handler is found in ArtifactsController.java, and is implemented as follows:

server/src/main/java/com/thoughtworks/go/server/controller/ArtifactsController.java

@RequestMapping(value = "/repository/restful/artifact/PUT/*", method = RequestMethod.PUT)
   public ModelAndView putArtifact(@RequestParam("pipelineName") String pipelineName,
                                   @RequestParam("pipelineCounter") String pipelineCounter,
                                   @RequestParam("stageName") String stageName,
                                   @RequestParam(value = "stageCounter", required = false) String stageCounter,
                                   @RequestParam("buildName") String buildName,
                                   @RequestParam(value = "buildId", required = false) Long buildId,
                                   @RequestParam("filePath") String filePath,
                                   @RequestParam(value = "agentId", required = false) String agentId,
                                   HttpServletRequest request
   ) throws Exception {
       // [1]
       if (filePath.contains("..")) {
           return FileModelAndView.forbiddenUrl(filePath);
       }
 
       // [2]
       JobIdentifier jobIdentifier;
       try {
          jobIdentifier = restfulService.findJob(pipelineName, pipelineCounter, stageName, stageCounter, buildName, buildId);
       } catch (Exception e) {
           return buildNotFound(pipelineName, pipelineCounter, stageName, stageCounter, buildName);
       }
 
       // [3]
       if (isConsoleOutput(filePath)) {
           return putConsoleOutput(jobIdentifier, request.getInputStream());
       } else {
           return putArtifact(jobIdentifier, filePath, request.getInputStream());
       }
   }

This code snippet is condensed for clarity, but three distinct steps can be identified:

• At [1], the value of filePath is validated to prevent path traversal attacks;
• At [2], various objects are created to keep track of the current job, artifact name, etc., and to format this data for the final stage;
• At [3], the artifact file is written to the local filesystem.

While the request parameter filePath is validated to prevent path traversal vulnerabilities at [1], that is not the case for the other request parameters, such as stageCounter.

Going deeper into the objects creation step ([2]), both a JobIdentifier and a StageIdentifier are instantiated. The role of these classes is to hold information about the CI job the incoming artifact is attached to, including values of the parameters filePath, stageCounter, and so on. This information is later used to craft the path the artifact will be written to.

When the call to putArtifact() is finally reached at [3], the JobIdentifier object is used to craft the destination path of the artifact:

server/src/main/java/com/thoughtworks/go/server/controller/ArtifactsController.java

private ModelAndView putArtifact(JobIdentifier jobIdentifier, String filePath,
                                   InputStream inputStream) throws Exception {
   File artifact = artifactsService.findArtifact(jobIdentifier, filePath);
   if (artifactsService.saveOrAppendFile(artifact, inputStream)) {
       return FileModelAndView.fileAppended(filePath);
   } else {
       return FileModelAndView.errorSavingFile(filePath);
   }
}

Finally, saveOrAppendFile() writes the file on the local filesystem:

server/src/main/java/com/thoughtworks/go/server/service/ArtifactsService.java

public boolean saveOrAppendFile(File dest, InputStream stream) {
   String destPath = dest.getAbsolutePath();
   try {
       LOGGER.trace("Appending file [{}]", destPath);
       try (FileOutputStream out = FileUtils.openOutputStream(dest, true)) {
           IOUtils.copyLarge(stream, out);

The final destination path, destPathis based on stageCounter, which is not validated: attackers can write files outside of the intended artifact directory. 

Exploitation Challenges

When dynamically stepping through the code, several exploitation constraints arose:

• The name of the resulting file is fully controlled, but the file is written in a sub-folder whose name is not controlled and is based on the current job’s name; 
• filePath can be empty, in which case the resulting file will be named with the current job’s name;
• When submitting a ZIP file, it will be safely extracted under a folder named based on the current job’s name.

Because of these restrictions, my team and I didn't find a way to gain arbitrary code execution without another intermediary step, even with a powerful exploitation capability like this one. (Did you? Let us know!). Since the final part of the destination path is based on the job name, attackers could use the Cross-Site Scripting vulnerability to force administrators to first create a job whose name is the destination they want to write to.

To exploit this vulnerability, the next step is to identify files and folders that are writable by the user under which the GoCD server is running and that may have a security impact if created or modified. Attackers usually try to target configuration files or directories where plugins can be installed, but GoCD does not automatically reload them upon new changes. 

I used the debugging tool strace to identify files that are accessed when browsing the GoCD interface and noticed that the GoCD java processes tried to load Ruby (ERB) templates:

[pid  2583] stat("/go-working-dir/work/jetty-0_0_0_0-8153-cruise_war-_go-any-/webapp/WEB-INF/rails/app/views/shared/error.en.html.erb", 0x7fb3fffe5ee0) = -1 ENOENT (No such file or directory) <0.000052>
[pid  2583] stat("/go-working-dir/work/jetty-0_0_0_0-8153-cruise_war-_go-any-/webapp/WEB-INF/rails/app/views/shared/error.en.erb", 0x7fb3fffe5ee0) = -1 ENOENT (No such file or directory) <0.000274>
[pid  2583] stat("/go-working-dir/work/jetty-0_0_0_0-8153-cruise_war-_go-any-/webapp/WEB-INF/rails/app/views/shared/error.html.erb", {[...]}) = 0 <0.000213>

While intriguing at first, this behavior can be explained by the presence of a Ruby On Rails application exposed under /go/rails/. When reaching non-cached pages of this subsystem, the Ruby On Rails rendering engine searches for templates at several locations: here, views/shared/error.en.html.erb, views/shared/error.en.erb and views/shared/error.html.erb. 

Creating one of them (e.g. /go-working-dir/work/jetty-0_0_0_0-8153-cruise_war-_go-any-/webapp/WEB-INF/rails/app/views/shared/error.en.html.erb) and browsing an invalid page below /go/rails/ loads this template renders its contents and grants arbitrary code execution. 

Patch

This issue was addressed by improving the validation of URLs and branch names in two commits on ArtifactsController.java:

• c22e042: the new method isValidStageCounter() ensures that stageCounter is a positive integer by using Integer.parseInt() in POST and PUT handlers.
• 4c4bb47: the same method is applied in the GET handler.

CVE-2021-43286 — Argument Injection in External SCM Invocations

By exploiting the Stored Cross-Site Scripting, attackers could also force administrators to create a new pipeline or configuration repository. This new repository would be cloned automatically by the server using external tools: for instance, referencing a Git repository will invoke the system-wide git command. 

This logic is implemented in domain/src/main/java/com/thoughtworks/go/domain/materials/. The method checkConnection() of classes is called when the Test Connection button is clicked or when a repository is created:

For Git, it is implemented as follows in GitCommand.java: 

domain/src/main/java/com/thoughtworks/go/domain/materials/git/GitCommand.java

public void checkConnection(UrlArgument repoUrl) {
   final String ref = fullUpstreamRef();
   final CommandLine commandLine = git().withArgs("ls-remote").withArg(repoUrl).withArg(ref);
   final ConsoleResult result = commandLine.runOrBomb(new NamedProcessTag(repoUrl.forDisplay()));

   if (!hasExactlyOneMatchingBranch(result)) {
       throw new CommandLineException(format("The ref %s could not be found.", ref));
   }
}

If you remember the previous post about the PHP Supply Chain Attack on Composer, you have probably already identified the vulnerability:  the variable repoUrl is user-controlled, its format is not validated, and it is concatenated into the command line. withArg() takes care of quoting the repoUrl value, which mitigates the risk of a command injection but does not prevent attackers from adding unintended arguments with the prefix --.

The combination of three factors leads to a best-case scenario for exploitation: 

• An argument can be added without character set restriction;
• git ls-remote requires a positional argument, and the server will always add refs/heads/master in the call;
• git ls-remote implements --upload-pack, an option to specify the path of the executable git-upload-pack on remotes.

Using --upload-pack=... in the URL field will result in the execution of the following command:

The refs/heads/master is the first positional argument: it forces git to treat it as a repository location. The value of the injection option --upload-pack has the specificity to be invoked as an external command even in the case of local repositories. As an example, using --upload-pack=”$(id>/tmp/id)” in the URL field confirms that attackers can gain arbitrary command execution:

bash-5.0$ ls -alh /tmp/id
-rw-r--r--    1 go       root          40 Oct 27 15:35 /tmp/id
bash-5.0$ cat /tmp/id
uid=1000(go) gid=0(root) groups=0(root)

My team and I gave the focus on git, but note that other handlers (SVN) were also vulnerable.

Patch

This issue was addressed with the commit 6fa9fb7, in which developers added stronger validation on user-controlled values and started using the end-of-options delimiter -- standardized by POSIX.

Timeline

Summary

In the previous blog post, I described a critical vulnerability that allowed unauthenticated attackers to get remote access to any GoCD installation. In this blog post, I described three additional vulnerabilities that could have been used by attackers to compromise a GoCD instance and to take over the underlying server.

I highly recommend that all users running GoC upgrade to the latest version (>= 21.3.0) since it includes patches for all the vulnerabilities I presented so far. 

My team and I would like to thank the GoCD Security Team, which has been exceptionally responsive in the disclosure process. They reacted very quickly and worked with us to patch the vulnerabilities efficiently.