EKS Security Checklist: 10 Best Practices for a Secure Cluster

Kubernetes is riddled with security challenges. Inevitably, the same goes for managed Kubernetes services like Amazon Elastic Kubernetes Service (EKS). The best way to tighten your cluster security is to implement practices that have become industry standards recommended by the Kubernetes community. Here are the ten most EKS security tactics every team needs to protect its clusters.

What Exactly Is Amazon EKS Security About?

Amazon EKS is one of the most popular managed Kubernetes services out there. It allows teams to orchestrate their containers via Kubernetes without installing and operating the Kubernetes control plane or the infrastructure required for Kubernetes clusters to run.

Since EKS is a service delivered by AWS, we can start the conversation about security with the shared responsibility model. In general, AWS is responsible for managing the security of its cloud services, while the customer oversees the security of the workloads.

AWS manages the Kubernetes dashboard and control plane via EKS, including all the infrastructure services used by the cloud provider to deliver a safe Kubernetes environment. 

Self-managed workers and EKS cluster configurations like IAM, pod, runtime, network security, worker node scalability, and container image components all fall to the customer’s plate.

Client-side security includes data security, upgrades and patches for worker nodes, and secure configuration for everything, starting with the data plane and nodes and ending with containers and operating systems. Customers also need to configure security groups allowing the EKS control plane to communicate with virtual private clouds (VPCs) in a secure manner.

AWS users get support from the cloud provider around EKS security in the form of:

• User guide to security,
• EKS Best Practices Guides. 

AWS Built EKS Security Features Into Its Managed Kubernetes Service

Here are a few built-in EKS security features every AWS user gets if they decide to run their clusters using this managed Kubernetes service.

AWS Secrets Manager

In Kubernetes, you can create key/value pairs and bring them to applications running inside your pods. If they contain any sensitive data, you can use Secret Store, which is a Container Storage Interface driver implemented by AWS Secrets Manager (as well as AWS Parameter Store).

AWS Secrets Manager gives you a central place for storing and managing Kubernetes secrets. The AWS Secrets and Configuration Provider (ASCP) plugin allows users to deal with legacy Kubernetes workloads that received secrets via ETCD. You can also apply IAM policies to define which pods can access the secret.

Identity and Access Management

Identity and Access Management (IAM) is a helping hand to every admin looking to control access to AWS resources. IAM administrators can set who can sign in and have permissions to EKS resources based on specific policies. 

Users get credentials for authentication and authorization of Kubernetes resources. The idea is to grant service users access only to the features they need to perform their job.

Logging and Monitoring

AWS offers access to CloudWatch, which logs stores diagnostic and audit logs from the control plane. Every EKS control plane gets its log group. It’s key that you monitor these logs to discover security and production issues on time. There’s also AWS CloudTrail, which records all the EKS activity and captures API calls made by users, roles, AWS services, or EKS console requests.

10 Best Practices for EKS Security

1. Isolate Kubernetes Nodes

Avoid exposing your Kubernetes nodes directly to public networks. Ideally, such nodes should be located on a separate network and have no direct connection to the general corporate network, if possible. 

How to make that work? Keep Kubernetes control and data traffic isolated. Otherwise, they will both end up flowing through the same pipe. Open access to the data plane implies open access to the control plane, and that’s bad news for EKS security. Configure nodes using an ingress controller and set it to only let in connections from the master node through a specified port in the network access control list (ACL).

2. Strengthen Authentication and Authorization

Integrating Kubernetes with a third-party authentication provider is a smart move. That way, you get an extra layer of security features – for example, multi-factor authentication. 

For secure control plane access, don’t manage users at the API server level but use the AWS Identity and Access Management (IAM) solution. Pick OpenID Connect (OIDC) alongside an SSO provider you got used to if you can’t get CSP IAM.¹ 

3. Take Advantage of Kubernetes Role-Based Access Control (RBAC)

Another access-related best practice for EKS security is using RBAC to define who has access to the Kubernetes API and what permissions they have. In Kubernetes 1.6 and higher, RBAC usually comes enabled by default. Given that Kubernetes combines authorization controllers disable the legacy Attribute Based Access Control (ABAC) when turning RBAC on.

It’s a good idea to choose namespace-specific permissions over cluster-wide permissions. Even when debugging, avoid granting cluster administrator privileges. Otherwise, your container security might become compromised.

4. Avoid Keeping Secrets in an Environment Variable

Ensure that your objects use a secret in an environment variable because other parts of your system can easily access environment variables. Use secrets as files or benefit from secretKeyRef to minimize potential threats.

5. Don’t Run Containers in Privileged Mode

If your deployment has containers running in privileged (root) mode, it allows the container to access important host resources. This may lead to security issues. Avoid running containers in privileged mode or turn the podSecurityPolicy on and set the privileged parameter to false. This will ensure that containers aren’t able to run processes that require root privileges on the host.

6. Don’t Share the Host’s IPC or Network Namespace 

Look at your pods and see if they share the host’s IPC or network namespace. Sharing namespaces for pod and host interprocess communication is dangerous because it may open access to shared information. For this reason, pods should never be allowed to access the host namespaces. 

Sharing pod and host network namespaces allows network access to the host network from the pod. This breaks network isolation. Set the hostNetwork parameter to false in PodSecurityPolicy and sleep better at night, knowing that your cluster is protected.

7. Disable NET_RAW

If your Kubernetes containers don’t drop the NET_RAW capability, you might enable a wide range of networking exploits from within the cluster. To ensure EKS security, use Policy Enforcement solutions like Open Policy Agents, Kyverno, or Kubernetes Pod Security admission controller to follow best industry practices.

Set drop for ALL or NET_RAW capabilities in the pod’s securityContext definition to make sure that the NET_RAW capability is disabled. [2, 3]

8. Check Unsafe /Proc Mount 

Deployments with unsafe /proc mount (procMount=Unmasked) allow others to bypass the default masking behavior of the container runtime. If you set a container with an Unmasked /procs mount type, you may expose the host information to the container, resulting in a potential data leak or container escape. Set procMount=Default to make sure that the container doesn’t expose any parts of /proc.

9. Don’t Use the Root File System for Container Security

If your containers run without a read-only root file system, prepare for trouble. Using a read-only file system can prevent all kinds of malicious binaries from writing to a system or a system takeover by attackers. Ensure containers use only the read-only file system and set the readOnlyRootFilesystem to true in Pod securityContext definition. 

10. Build a Rolling Update Strategy

Finally, to keep your EKS security tight, develop a rolling update strategy. Rolling updates let deployment updates minimize your application downtime by incrementally updating pod instances with new ones. Check out this page in the Kubernetes docs for more details.

Another point is having a vulnerability scan at runtime, as there are supply chain attacks, so you need to see what really got to your cluster, even if you scanned your deployment artifacts in the CI/CD stage. Generally, agent-based security solutions are good or even better than “agentless” ones.

The Kubernetes ecosystem is evolving, and its security is no exception. As new threats emerge and problems come to light, engineers are forced to keep up with a lot of things – which takes a lot of time and effort.

Another challenge they experience around EKS security is the prioritization of security issues – depending on the application’s size, prioritizing may become very time-consuming. Automation can accelerate this, winning engineers’ time back for other tasks.