Hardcoded Secret at the Heart of the Dell Compellent VMware Vulnerability

In August, Dell disclosed vulnerability CVE-2023-39250 where "A local low-privileged malicious user could potentially exploit this vulnerability to retrieve an encryption key that could aid in further attacks." This actively affects Dell Storage Integration Tools for VMware (DSITV) customers. Learn how to protect yourself from this vulnerability and some tips on preventing similar mishaps in your codebases.

How Do I Mitigate This as a Dell Compellent Customer?

Before diving into what happened, if you think you might be affected, we encourage you to start the investigation and mitigation process as soon as possible. According to the report released by Dell, all users of DSITV should follow these workaround and mitigation steps:

1. Please follow the instructions in the Dell Storage Integration Tools for VMWare Version 6.0 Administrator’s Guide to change the default root password of all current and new appliances using Compellent DSITV.
2. Update the password to the VMware vCenter.
3. Do not create additional DSITV users; if additional users have already been created, remove those users.
4. Do not change file/folder permission levels for DSITV; ensure that “/opt/dellcompellent” requires root level to access.

Some online reports have questioned how the proposed workaround would prevent a local user from extracting the affected key. We still urge all users to take the prescribed actions and make plans to implement the patch that Dell has promised to release by November 2023.

What Happened

On August 11, 2023, Dell published a Security Update disclosing a shared encryption key used across all installs of their Compellent Integration Tools for VMware. This shared password means that once cracked, the same key would allow attackers to decrypt stored vCenter admin credentials, allowing the highest-level access to the system.

Dell describes their Compellent tooling as a line of enterprise storage systems "actively manage data, based on data access patterns, at a highly granular level using built-in intelligence and automation features." Compellent integrates with VMware vCenter, a popular virtual machine management tool relying on a Dell encrypted configuration file for authentication. This is where the vulnerability manifests.

This vulnerability was first reported to Dell by security researcher Tom Pohl on April 11, 2023. However, it took until August for Dell to confirm the results and disclose the scope of the issue.
At the core of the issue is a shared Advanced Encryption Standard, AES, encryption key. Tom's team at LMG Security found a JAR file in the Dell Compellent software directory that they could decompile. When they did, they discovered a hardcoded static AES key. Further, the research showed this same key would be used across all installs of the software.

If an attacker discovered this key, they could unencrypt all other vCenter credentials stored in the program's configuration file.

What You Can Do to Prevent Similar Issues in Your Projects

While this particular vulnerability affects Dell and VMware customers, the root causes of the issue are one that can potentially affect projects of any size or maturity. Here are some wider lessons we can all apply to stay safer in our work.

Replace All Default Credentials

In the research done by LMG Security, they gained access to production servers containing the credentials by successfully attempting to use the default server username/password, `admin/admin`. This is not that surprising given that, according to some reports, up to 40% of IT security leaders don’t change default admin passwords.

Attackers generally will always try the path of least resistance first. If a simple script that attempts all known default username/password pairs grants them access, they will forego more laborious and time-intensive invasion techniques. Malicious actors have limited time and resources, so they are always looking for a quick win, especially early in an attack.

Don't Hardcode Plaintext Passwords

Unfortunately, at the heart of this matter is the all too common issue of a hardcoded secret. If you have read our State of Secret Sprawl report, you already know this is a growing problem affecting 1 in 10 committers to public GitHub repos, totaling 10 Million leaked secrets in 2022 alone.

In Dell's case, they were relying on compiled code to be enough to stop someone from finding their AES password. As proven by the research team, decompiling is straightforward, and many tools make this easy to accomplish. If at any time you can read your credentials in plain text, assume that anyone who gains access to your code can as well.

The most effective way to prevent hardcoded credentials from entering your project is by catching them early in the process, ideally before any commits are made at the local developer's machine. Using tools like ggshield, the GitGuardian CLI, developers can set up pre-commit git hooks that automatically check for secrets before the commit is added to the git history.

If credentials still make it into your code, getting alerted immediately is your next line of defense.

The Need for Unique Passwords

The discovery of an encryption key for a single system was already bad enough, but in this case, it was a common key used across all installs of the product. While this might be a convenient path to quickly roll out integrations, as this disclosure shows, this is ultimately not the safest path.

If at all possible, all credentials should be tightly scoped, only allowing access to a single service or encrypted data source. If an attacker does get access to a single installation or a specific service, then you should work to ensure the potential blast radius will be as small as possible.

Monitor Access

When in doubt about who has accessed what within your system, that is when you can rely on your logs. If you see new and unknown IP addresses getting admin-level access, then you likely have an issue. Examining logs only after an incident can only help so much. Ideally, you shouldget alerts as soon as something suspicious happens. Turning on alerts and fine-tuning notification settings can help you make the most out of logging and data tools you have already invested in.

In addition to the defensive steps mentioned above for monitoring your repos for hardcoded credentials, you can actively set traps for attackers. We know that attackers are looking to exploit any and all credentials that might grant access.

Stay Safe as You Integrate

This incident is ongoing, as the patch to address this issue is not expected to be released until November. However, there is still a lot we can take away from what we already know. Integrating tools is an ongoing challenge, especially at scale. Investigate safer methodologies that lock encryption keys in vault systems, hardware security modules, or any other solution to keep vital passwords away from prying eyes with access to your code.

But even with the best solutions, implementation effectiveness can vary, and human error will always be with us. You can take a more proactive defensive posture, leaving decoy credentials for malicious actors to exploit at their own peril.