Four Ways to Keep Kubernetes’ Secrets Secret

We have talked a lot about the speed at which DevOps innovation has moved and how security has consistently struggled to catch up. Kubernetes is quickly putting this idea to shame and stretching security teams to their limit. In just five short years, Kubernetes has exploded in usage, but security wasn’t always at the front of everyone’s minds.  

One of the most shocking recent Kubernetes developments was the discovery of the most severe Kubernetes vulnerabilities ever, CVE-2018-1002105, which we discuss further here.  The silver lining here is that the vulnerability led to the realization that Kubernetes developers need better security practices.  To be fair, security, as with DevOps, is a process of continuous improvement. In this blog, we will discuss best practices for securing Kubernetes.

1. Monitor and Control Access to Kubernetes Containers and Pods

Proper secrets management is the key (pun definitely intended) to controlling human and non-human access to containers and pods within Kubernetes.  Secrets – such as passwords, SSH Keys, certificates, API tokens and so on – grant container access to anyone who knows them.  

You may also like: Kubernetes Secrets Management.

Organizations become vulnerable when applications or application developers mishandle secrets by leaking them to logs or exposing secrets in application code accessible to other developers and staff or, even worse, the general public or in GitHub or some other source code management (SCM) tool. I am going to mention GitLab, Mercurial, and CVS as other SCM examples, so they don’t feel left out.  

This breach of trust is further exacerbated by the unique ability of secrets to grant access to additional secrets — highlighted by the secret zero problem — and additional privileges. A few mishandled secrets can be leveraged to escalate access and privilege, allowing attackers to expand a small breach into a massive attack that spiders an entire network.

To limit your exposure and the potential damage of mishandled secrets, encrypt and tightly control access to secrets. Access to secrets should be authorized based on a security policy that works for highly elastic environment like Kubernetes. Role-Based Access Control (RBAC) is a good way to authorize new containers, as they spin up because it doesn’t require any agents or prior setup for the new containers.

Secrets should be rotated on a regular basis to keep access out of the hands of those who don’t need it. Ephemeral secrets, dynamic secrets, and just-in-time access are all different ways to make sure the same secret can’t be used later by an attacker. There are pros and cons to each method that we won’t cover in this blog, but you can read more about the differences here. What is important is that secrets are changed on a regular basis.

To limit unauthorized access and prove compliance, it is important to audit access to critical systems.  Central audit trails are a good way to provide visibility into critical security events.

2. Strongly Authenticate Kubernetes Container Requests for Secrets

RBAC and other good secrets management security policies ensure that access is properly authorized so that access doesn’t fall into the wrong hands, but this isn’t good enough if the secrets provider gives the secret to an imposter.  Some secrets providers have some sort of secrets-based authentication of clients requesting secrets, but what is used to secure those secrets, another secret?  Thus, we return again to the secret zero problem.

Strong Kubernetes security requires strong Kubernetes authentication of client requests for secrets. There are a number of approaches to this, but the best approach is to avoid the secret zero problem by working with Kubernetes, the underlying container orchestration system running your applications, to automatically authenticate containers requesting secrets with strong multi-factor authentication.

To authenticate secret requests, the solution should use multiple attributes only available to trusted containers, such as certificates generated with the Kubernetes APIs, securely random UUIDs, cryptographic keys, role, name, and other automatically generated/configured attributes in the orchestration system of choice.  

The presence of just one factor presented by an application should not be enough to authenticate container requests. Multiple factors should be required for authentication.  This way, a stolen credential will have no value if the attacker using it can’t be authenticated.

3. Isolate Kubernetes Applications from Secrets

Developers don’t really want to manage secrets, but they also want the freedom to do their jobs and get access to the systems their application code needs, like databases, SSH servers, and other critical services. Usually (unfortunately), this involves handling secrets from secrets providers. Also, not all applications are built to securely handle secrets, resulting in accidental leaks to logs and debug records — even ephemeral secrets can be leaked, and they can be valid for minutes or days (depending on user configuration).

Building applications with the Secretless pattern stops the application from leaking secrets — since they can’t leak what they don’t have — while making it easier for developers to get access to protected resources.  Check out the Secretless Broker open source project as an example of this.

Secretless Broker

Secretless Broker, launched earlier this year, is an open source project that simplifies how containerized applications running in Kubernetes securely access databases, HTTPS-based web applications, and servers. A new SDK was recently added that enables developers to write new service connectors to extend the range of databases and other external resources accessed by applications using Secretless Broker.

With Secretless Broker, when an application needs to securely access a resource, the app simply makes a local connection request to Secretless Broker. Secretless Broker then automatically authenticates the app, fetches the required credentials from the secrets provider, and establishes a connection to the database or other resource. With Secretless Broker, the application no longer has access to credentials, preventing them from being accidentally leaked or exposed by the application and reducing the attack surface.

This approach offers two major simplifications for developers. First, it eliminates the need for the developer to write API calls to fetch the access credential or secret from Kubernetes secrets (or any other secrets store) and, second, it removes the need for the application to directly handle secrets.

Secretless Broker provides development teams deploying applications in Red Hat OpenShift and Kubernetes environments with a simplified option for applications to securely access MySQL and PostgresSQL databases. The SDK enables developers to add support for additional databases and resources.

4. Centralize Secrets Management

There are a number of secrets management tools available within various Cloud platforms like AWS and DevOps tools like Ansible and Puppet and, yes, orchestration platforms like Kubernetes (Kubernetes Secrets). Within most environments, more than one of these tools will be required to manage all secrets.

This creates blind spots within an organization’s access management picture and other difficulties since these tools are not interoperable or fully featured for secrets management. Centrally managing secrets is also key to fully auditing access, centralizing audit records in the process. Distributing secret management responsibilities across multiple tools creates what is known as “Security Islands.”

Security Island
A tool or platform that comes with some level of its own security components built-in to provide some ability to manage secrets, access control, audit, compliance and the equivalent, but which does not facilitate interoperability with other tools or aggregation of security policies, management, and audit data. Also, various tools have varying levels of security capabilities and maturity.

Using a centralized secrets management solution makes audit, access control, and secret management more manageable by giving organizations a centralized view of their overall Kubernetes security landscape, allowing them to understand and control how individual containers and services interact with each other. We recommend Conjur open source.

Open Source Secrets Management for Kubernetes and More

Conjur Open Source helps eliminate islands of security by providing an interface for consistently and securely authenticating, controlling, and auditing non-human access across tool stacks, platforms, and cloud environments via robust secrets management capabilities. Conjur helps organizations secure secrets such as passwords, SSH keys, certificates, and API keys and implement management best practices, including strong authentication, least privilege, role-based access control (RBAC), credential rotation, management and audit. Try the quick start tutorial here.

With Conjur, secrets can be pushed to native Kubernetes Secrets store. CyberArk’s out-of-the-box integration with Kubernetes Secrets strengthens the security of the native Kubernetes secrets mechanism by enabling CyberArk to centrally manage secrets without requiring any changes to code or further involvement from developers.

One challenge that comes with using the native secret stores of various tools and platforms in development and production environments is that not only are credential management, rotation, audit, and other similar processes limited and inconsistent, but the ability to securely share secrets across tools is also limited.

For organizations using Conjur Open Source to manage application secrets and other credentials, secrets are pushed to the native Kubernetes secrets store based on Conjur policies. Conjur’s management of Kubernetes is completely transparent to developers, since no changes to code are required. The application code simply accesses the secrets in the Kubernetes secrets store exactly as it did before.

For additional information on the Conjur Kubernetes Secrets integration read our technical documentation and GitHub.

TL;DR

1. Monitor and control access to Kubernetes pods and containers via robust secrets management.
2. Use strong identities for human and non-human access to authenticate anyone requesting secrets.
3. Isolate applications and application developers from the burden of safely handling secrets.
4. Centrally control, enforce, monitor, and manage access with centralized secrets management across applications, clouds, and tools.

Further Reading

• The Complete Kubernetes Collection [Tutorials and Tools].
• Monitoring Kubernetes.
• Securing Your Kubernetes Deployment.