What We Learned About Secrets Security at AppSec Village at DEF CON 32

If you grew up in the US, chances are you have a memory of going to summer camp. Even if you didn't attend one yourself, the camp experience of going away from home, learning all sorts of arts and crafts, meeting new best friends, and going on memorable adventures is baked into pop culture and media. Every August, the largest hacker summer camp on earth takes place in the heat of Las Vegas. This year marked the thirty-second iteration of DEF CON.

DEF CON can be hard to explain without experiencing it. Yes, there are speaking tracks, official workshops, and multiple capture-the-flags (CTFs), but there is so much more. No other conference contains so many sub-conferences and community-led events. Even attendees who have been going for years say they still don't think they have experienced everything on offer.  

While there is no official number released by the organizers, past events have ranged from 25,000-30,000 attendees. Each and every attendee brings with them a love for tech and plenty of knowledge to share. The hallways are full of people hacking on hardware, writing software, making music, sharing stickers, and making all the DEF CON fun happen.

The Villages

There were 33 villages contained within DEF CON 32. These cover a wide diversity of interests, from Aerospace hacking, Social Engineering, and Misinformation to Red Teaming and AppSec. Each village is independently organized and offers a unique set of talks, workshops, and hands-on activities for attendees to interact with the specific area of their focus.

AppSec Village was a valuable space within the larger hacking and security community that focuses on defending the applications that power all our lives. Your author was one of the fortunate people who helped organize and run AppSec Village 2024.

Spot the Secrets 

Back at AppSec Village at RSA Conference 2024, GitGuardian revealed our Spot the Secrets, a card-based game that simulates the experience of manual code review for finding plaintext credentials. Players are asked to race to find all the hidden API keys, passwords, and other plaintext secrets attackers could use to gain further access to a collection of code samples, Jira tickets, log files, and Slack messages. 

Spot the Secrets at AppSec Village, edited to account for DEF CON photo rules

At DEF CON, over the course of 2 days, more than 120 people came through our POD at AppSec Village to experience this exercise for themselves. We even had a leaderboard where the fastest players, who made the fewest mistakes, won special swag prizes.

The Most Common Secrets Security Questions

As people started examining the cards containing code, we got a wide array of questions. We believe it would be beneficial to share some of the questions in case you, too, have not heard answers to these concerns yet.

Here are the top 11 most common questions we heard and our replies.  

1. What Is a Commit Exactly?

A commit is a unit of work in Git, the version control system used by over 97% of developers globally. It is a snapshot of the file system at a given moment of time, encompassing all the work the developer has done since the last commit. If you line up commits one after the other, you get the Git history, which lets you examine each change made incrementally in the life of the codebase. One of the secret security challenges of using Git is that the history is a shared permanent record. If a secret is added to one commit and removed in the following commit, the commit with the secret is still preserved.

2. How Can You Tell If Something Is a Secret?

A secret is any credential that gives direct access to a system or data. These can take the form of API keys, passwords, certificates, and tokens, to name a few. Part of the reason they are hard to spot is that they are used in a multitude of ways within code and from the command line. For example, inside a configuration file, a specific variable might be set for an API key. But for systems like Slack, the API token is embedded in the URL itself, meaning you would need to be familiar with that system to know it is a secret.

Many systems use long, unique numbers in their user interface URLs, which can very closely resemble inline tokens. Figma URLs, for example, may seem to contain a base64 encoded string in the path. However, if you follow a random Figma link, you will be met with a login screen, as you need to be an authenticated user to connect to that page. Since it does not automatically grant system access but merely identifies the page location, an attacker with no foothold in Figma would return the "No Access" error and move on. This is another reason to embrace tooling to identify secrets, as there are platforms that can quickly tell the difference between a URL that contains an authorization token and one that just seems like it might.

3. What Does Valid Mean?

"Valid," in our use, means that a particular credential still works to grant access to the related system or data. If found by an attacker, a valid secret could be immediately used. While non-valid secrets can give an attacker insight into the overall architecture and interconnected systems, only valid secrets offer them a direct path into those resources. These types of lateral movements can be hard to detect as well.

4. How Am I Supposed to Know Which Secrets Are the Right Ones to Look For?

The best answer is to find and remove any valid secrets. However, as Spot the Secrets proves, this is a challenging problem to solve without the right tools. Manual code reviews or tools relying solely on simple pattern matching can lead you to spend time remediating already invalidated secrets rather than focusing on valid ones. Expired keys are not much of a threat, so efforts to eliminate them should take a lower priority to any valid credentials you discover.

5. What’s the Difference Between a Secret and Just Poorly Written Code?

Attackers are likely not reading your code most of the time; they are scanning for specific ways to gain wider access as quickly as possible. Poorly written code can contain many vulnerabilities that an attacker can use, especially around Infrastructure as Code configuration. However, exploiting these flaws requires an attacker to take the time to understand the flaw and execute a specific attack against such issues rather than just leveraging plaintext credentials.

Variable names and usernames can be used as part of a more elaborate attack. For example, usernames and emails found in code can be used in password spray attacks or brute force attempts, but those are much easier to detect than direct secrets usage and very time-consuming for the attacker.

6. But What Do I Do with It Once I Find a Secret?

First, stop and breathe. It happens to the best of us, so don't get too upset if you find a leaked secret. It is good to act with urgency, but don't panic and revoke a found secret without knowing what will happen. This can cause a multitude of other issues.

We recommend following the path of:

1. Understand what the secret unlocks.
2. Figure out how critical any associated data or system is.
3. Check your logs for unauthorized use of the secret. 
4. Determine if any data or service access has been leaked.
5. Find out what will break when you rotate the secret.
6. Rotate the secret and store the new credential safely.
7. Fix any broken workflows or production deployments.
8. Review the incident and create an action plan to avoid further secret leakage.

Read more about the remediation process in the article, "What to Do if You Expose a Secret: How to Stay Calm and Respond to an Incident."

7. My Team Always Rewrites the Git History When a Secret Is Discovered. Is That All We Need to Do?

Unfortunately, no. While we love the Git cleanup step of the remediation process, it is truthfully the final step and is seen by many organizations as optional. We recommend rotating any secrets ever exposed as plaintext (see the above link for remediation process advice).

8. I Just Remove the Repo From Public. Isn’t That Enough?

We honestly wish this was the case, but unfortunately for all of us, once it is exposed on GitHub publicly, it should always be considered compromised. GitGuardian scans every single new commit and isPublic event that happens across the GitHub public API.

This is how we build our annual State of Secrets Sprawl report and is the basis for the GitGuardian Public Monitoring offering. While our intentions are to alert committers that they did something potentially dangerous, we are not the only actors monitoring this public API. You should always assume there is a copy of any code or files you ever pushed publicly, made by someone you don't know, and stored somewhere else you don't know about. We always recommend rotating any potentially exposed secrets.

9. Why Are False Positives a Problem?

In our Spot the Secrets exercise, we ask users to find all the secrets placed inside 100 possible commits, Jira, Slack, and Log cards, enforcing a time penalty for each false positive. Any card where the player falsely thought a card contained a secret when it did not would incur a 10-second penalty per mistake.

While in the limited scope of the exercise, it might seem like a good thing to be extra cautious and flag things that did not contain secrets, in all reality, it wastes valuable time, both for you, the reporter, and for anyone who is tasked with working on a solution.

Any tool or process that reports too many false positives leads to alert fatigue. Users are flooded with alerts and start to ignore them. This leads to distrust of the tooling and inconsistent use. It can also jam up the remediation process overall and mean that some reported true positive secrets are not taken care of in time.

10. What Does the File Name Have to Do With Identifying a Secret?

Simply put: context. Generic secrets can be especially hard to find through pattern matching alone. If there is a long string in a markdown file, our findings point to a high likelihood that it is an example password, so you would unlikely need to deal with it. If, on the other hand, that same string appears after the string password= in a file named project.env, then there is a much higher chance you have a real secret on your hands.

The file name is just one of the elements we consider with Pre- and Post Validation in the GitGuardian Secrets Detection Engine and is one of the factors that helps us eliminate false positives with the internal to the platform FP remover.

11. Do People Really Do Manual Code Reviews to Find Secrets?

Yes. A full 27% of IT decision-makers we surveyed said they rely on manual code reviews to remediate secrets sprawl, even though 75% said they had been affected by a secret leak.

Spot the Secrets was designed to show the issues with relying on humans to spot these issues. Machines are much better at pattern matching throughout thousands or millions of lines of code.

Learning Together Is What DEF CON Is All About

We learned plenty of other lessons at DEF CON, including staying hydrated, getting at least a few hours of sleep a night, and keeping ourselves indoors when it is over 110°F/43°C outside.

There were so many excellent talks on the schedule; we can't wait for the videos to become available to catch up on the content we missed while helping people learn about Secrets Sprawl. Along the way, we learned a lot about how security folks, developers, and people in general see this problem and think about solutions. We at GitGuardian hope to incorporate those lessons to make solving this problem more accessible and easier for us all.

If you have never considered going to DEF CON, we highly recommend it, if for no other reasons than to take part in AppSec Village and to find your tribe at hacker summer camp.