Lessons On Secrets Security From Datadog Research

Recently, Datadog released their report on attacker techniques that they saw from malicious IP addresses that affected multiple AWS environments from late 2023 through early 2024. They revealed findings from two different types of attack, each revealing a different level of sophistication.

Thankfully, no production environments were compromised, nor was any customer data accessed in these attacks. We are grateful to the Datadog team for their transparency in the report, as there are some important lessons we can learn from how attackers commonly behave from these attempts.  

A Tale of Two Attacks

In both types of attack described in the Datadog report, and just like the majority of all attacks, attackers gained an initial foothold using ill-gotten credentials. In this case, leaked IAM user keys were to blame. We highly suggest reading the Datadog article for an in-depth look, but here is a high-level summary of what happened. 

A Sophisticated Attack Meant To Evade Detection

In the first attack, after gaining initial access, the adversarial actor strengthened their foothold by creating new admin accounts, which would allow them continued access in case the initial access was terminated. Next, they focused on escalating privilege through account manipulation. After that, they looked through any service or bucket they could access to see if there was anything to leverage or steal, especially more credentials that allowed wider access. 

Ultimately, they tried to create new EC2 instances in a different region, which is when the attack ended unsuccessfully. If Datadog had not been monitoring for suspicious activities outside of their normal regions, this attack could have persisted until the bill came due. It is a good reminder that your attack surface is wider than you might initially suspect.

The Datadog team mapped this process out, summarizing one of the attacks with the following illustration. 

A Less Delicate Approach

The second type of attack took a lot fewer steps. The attacker got in and immediately began spinning up a lot of malicious infrastructure as fast as they could. Using randomized naming, they created many new clusters to run malicious code across 17 regions. This would allow them to deploy thousands of containers to execute malicious code. 

Just like in the first attack scenario, the attacker widened their foothold as fast as they could by moving beyond the initial region. Fortunately, the Datadog team was able to work with the AWS team to find the root of the problem, in this case, allowing IAM users with static credentials to perform such costly actions. Datadog has since updated its playbooks. 

What Can We Do to Better Defend Ourselves?

Even though no customers were affected or anything truly terrible resulted, the fact that these malicious actors could get so far into the Datadog infrastructure is alarming. It is a good reminder to us all to improve our security posture, especially around secrets. 

Leaked Long-Lived Passwords Continue to Be a Problem

Any password runs the risk of being leaked. If it is still valid 30, 90, or 365 days after it leaks, that is going to open you up to a lot of risk of a successful initial breach. If that credential is automatically rotated every 24 hours, the likelihood that it could be exploited in case of a leak drops significantly.  

For Root user passwords, it is a good idea to rotate passwords on a regular basis and regularly ensure your password reset path is secure as well.  For any other passwords, AWS has a clear path for using Secrets Manager and some simple scripting to auto-rotate secrets as often as you like. 

Always Remember the Principle of Least Privilege 

There are two things that are true about Root accounts on AWS:

1. They are powerful and, therefore, potentially dangerous.
2. They should not be used for anything beyond the highest level of administration. 

What you want to do instead is create users with limited permissions, scoped as tightly to the specific task as possible, following the Principle of Least Privilege.

If access is stolen or shared for one of these users, the attackers will have very limited powers and most likely would not have been able to get as far as the attackers did in Datadog's findings.   

No Password Is Better Than a Short-Lived Password

The most secure password is one that does not exist. On AWS and most other cloud providers, there is a clear and well-discussed path to eliminating most passwords through the use of IAM Roles. Assigning a user or application a role will grant them temporary access when it is needed. This approach also lets you get very granular with the permissions within defined roles, meaning access is only granted in tightly scoped circumstances. 

As the second type of attack showed, when an attacker creates a new user account, they don't assign them IAM roles. They give them passwords. That should give you some pause if you are still using long-lived passwords for your trusted users. 

Remove Outdated and Unneeded Credentials

At some point in your cloud application's history, it is likely someone used a long-lived password to allow needed access for the app to function. In the best of scenarios, they completely removed these credentials, revoked them, and adjusted the code to use IAM rules or a secrets manager. 

There is also the scenario that those old passwords are still in Git commits from the project's early days and are still valid.  Use secret scanning tools that can look at the whole Git history, and through every branch, to let you know so you can remedy the situation if necessary. There are many tools and platforms out there.

Get Alerts When Leaks or Breaches First Occur

In both of these cases researched by Datadog, the initial access was from a leaked credential. There are a number of ways an attacker can get a hold of these secrets, but based on what we have seen, they might have found them in public on GitHub. Ideally, you want to know as soon as they appear in public so you can act to protect your organization.

We recommend a combination of honeytokens and aggressive public monitoring. These are two different approaches but overlap and complement each other well. If a honeytoken is deployed to a private repo and that code is then leaked to the public, your honeytoken will go off pretty much immediately, letting you know you have an issue.

Learning From Research 

We are very thankful for the research by our partners on the Datadog team and other security professionals who surface findings of this depth. There is a lot we can learn from these findings, even if they did not result in successful headline-grabbing breaches or involve customer data leaks. Hopefully, your organization can apply these lessons and will never fall victim to these attackers.

Security is a journey, not a constant state you can permanently achieve.