Optimizing Kubernetes Clusters for Better Efficiency and Cost Savings

Optimizing resource utilization is a universal aspiration, but achieving it is considerably more complex than one might express in mere words. The process demands extensive performance testing, precise server right-sizing, and numerous adjustments to resource specifications. These challenges persist and, indeed, become more nuanced within Kubernetes environments than in traditional systems. At the core of constructing a high-performing and cost-effective Kubernetes cluster is the art of efficiently managing resources by tailoring your Kubernetes workloads.

Delving into the intricacies of Kubernetes, it's essential to comprehend the different components that interact when deploying applications on k8s clusters. During my research for this article, an enlightening piece on LinkedIn caught my attention, underscoring the tendency of enterprises to overprovision their Kubernetes clusters. I propose solutions for enterprises to enhance their cluster efficiency and reduce expenses.

Before we proceed, it's crucial to familiarize ourselves with the terminology that will be prevalent throughout this article. This foundational section is designed to equip the reader with the necessary knowledge for the detailed exploration ahead.

Understanding the Basics

• Pod:  A Pod represents the smallest deployable unit that can be created and managed in Kubernetes, consisting of one or more containers that share storage, network, and details on how to run the containers.
• Replicas: Replicas in Kubernetes are multiple instances of a Pod maintained by a controller for redundancy and scalability to ensure that the desired state matches the observed state.
• Deployment: A Deployment in Kubernetes is a higher-level abstraction that manages the lifecycle of Pods and ensures that a specified number of replicas are running and up to date.
• Nodes: Nodes are the physical or virtual machines that make up the Kubernetes cluster, each responsible for running the Pods or workloads and providing them with the necessary server resources.
• Kube-scheduler: The Kube-scheduler is a critical component of Kubernetes that selects the most suitable Node for a Pod to run on based on resource availability and other scheduling criteria.
• Kubelet: The kubelet runs on each node in a Kubernetes cluster and ensures that containers are running in a Pod as specified in the pod manifest file. It also manages the lifecycle of containers, monitors their health, and handles the instructions to start and stop containers directed by the control plane.  

Understanding Kubernetes Resource Management

In Kubernetes, during the deployment of a pod, you can specify the necessary CPU and memory—a decision that shapes the performance and stability of your applications. The kube-scheduler uses the resource requests you set to determine the optimal node for your pod. At the same time, kubelet enforces the resource limits, ensuring that containers operate within their allocated share.

• Resource requests: Resource requests guarantee that a container will have a minimum amount of CPU or memory available. The kube-scheduler considers these requests to ensure a node has sufficient resources to host the pod, aiming for an even distribution of workloads.
• Resource limits: Resource limits, on the other hand, act as a safeguard against excessive usage. Should a container exceed these limits, it may face restrictions like CPU throttling or, in memory's case, termination to prevent resource starvation on the node. 

apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: example-deployment
 spec:
   replicas: 2
   selector:
     matchLabels:
       app: example
   template:
     metadata:
       labels:
         app: example
     spec:
       containers:
       - name: example-container
         image: nginx:1.17
         ports:
         - containerPort: 80
         resources:
           requests:
             memory: "64Mi"
             cpu: "250m"
           limits:
             memory: "128Mi"
             cpu: "500m"

Let’s break down these concepts with two illustrative cases:

Case 1 (No Limits Specified)

Imagine a pod with a memory request of 64Mi and a CPU request of 250m on a node with ample resources—4GB of memory and 4 CPUs. This pod can utilize more resources than requested without defined limits, borrowing from the node's surplus. However, this freedom comes with potential side effects; it can influence the availability of resources for other pods and, in extreme cases, lead to system components like kubelet becoming unresponsive. 

Case 2 (With Defined Requests and Limits)

In another scenario, a pod with a memory request of 64Mi and a limit of 128Mi, along with a CPU request of 250m and a limit of 500m, finds itself on the same resource-rich node. Kubernetes will reserve the requested resources for this pod but enforce the set limits strictly. Exceeding these limits in the case of memory can result in the kubelet restarting or terminating the container or throttling its CPU usage if the CPU maintains a harmonious balance on the node.

The Double-Edged Sword of CPU Limits

CPU limits are designed to protect the node from overutilization but can be a mixed blessing. They might trigger CPU throttling, impacting container performance and response times. This was observed by Buffer, where containers experienced throttling even when CPU usage was below the defined limits. To navigate this, they isolated "No CPU Limits" services on specific nodes and fine-tuned their CPU and memory requests through vigilant monitoring. While this strategy reduced container density, it also improved service latency and performance—a delicate trade-off in the quest for optimal resource utilization.

Understanding Kubernetes Scaling

Now that we've covered the critical roles of requests and limits in workload deployment let's explore their impact on Kubernetes' automated scaling. Kubernetes offers two primary scaling methods: one for pod replicas and another for cluster nodes, both crucial for maximizing resource utilization, cost efficiency, and performance.

Horizontal Pod Autoscaling (HPA)

Horizontal Pod Autoscaling (HPA) in Kubernetes dynamically adjusts the number of pod replicas in a deployment or replica set based on observed CPU, memory utilization, or other specified metrics. It's a mechanism designed to automatically scale the number of pods horizontally—not to be confused with vertical scaling, which increases the resources for existing pods. The HPA operates within the defined minimum and maximum replica parameters and relies on metrics provided by the cluster's metrics server to make scaling decisions. It is essential to specify resource requests for CPU and memory in your pod specifications, as these inform the HPA's understanding of each pod's resource utilization and guide its scaling actions. The HPA evaluates resource usage at regular intervals, scaling the number of replicas up or down to meet the desired target metrics efficiently. This process ensures that your application maintains performance and availability, even as workload demands fluctuate.

The example below automatically adjusts the number of pod replicas within the range of 1 to 10 based on CPU utilization, aiming to maintain an average CPU utilization of 50% across all pods.

apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 metadata:
   name: example-hpa
   namespace: default
 spec:
   scaleTargetRef:
     apiVersion: apps/v1
     kind: Deployment
     name: example-deployment
   minReplicas: 1
   maxReplicas: 10
   metrics:
   - type: Resource
     resource:
       name: cpu
       target:
         type: Utilization
         averageUtilization: 50

Cluster Autoscaler

The Cluster Autoscaler can automatically adjust the size of cluster nodes so that all pods have a place to run and there are no unneeded nodes. It works by increasing the number of nodes during high demand when pods fail to launch due to insufficient resources, and it decreases the number when nodes are underutilized. The autoscaler estimates the necessity for node scaling based on Pod resource requests—pods that cannot be scheduled due to lack of resources will trigger the autoscaler to add nodes. In contrast, nodes that have been underutilized for a set period of time and have pods that can be comfortably moved to other nodes will be considered for removal. This ensures a cost-effective and performance-optimized cluster operation.

Conclusion

Optimization is not a one-time event but a continuous process. Rigorous load testing is essential to comprehend how an application performs under varying levels of demand. Utilizing observability tools such as NewRelic, Dynatrace, or Grafana can reveal resource consumption patterns. Take the average resource utilization from several load tests and consider adding a 10-15% buffer to accommodate unexpected spikes, adjusting as necessary for your specific application needs. Once you establish baseline resource needs, deploy workloads with appropriately configured resource requests and limits. Ongoing monitoring is paramount to ensure resources are being used efficiently. Set up comprehensive alerting systems to notify you of underutilization and potential performance issues, such as throttling. This vigilance ensures your workloads are not just running but running optimally. Organize your infrastructure by creating distinct node groups tailored to different application types—like those requiring GPUs or high memory. In cloud environments, smart utilization of spot instances can lead to substantial cost savings. However, always prioritize non-critical applications for these instances to safeguard business continuity should the cloud provider need to reclaim resources.