Preventing Advanced Persistent Threats in Kubernetes

The Sysdig Threat Research Team (TRT) published its latest Cloud-Native Security & Usage Report for 2024. The report sheds additional light on critical vulnerabilities inherent in current container security practices. This blog post delves into the intricate balance between convenience, operational efficiency, and the rising threats of Advanced Persistent Threats (APTs) in the world of ephemeral containers – and what we can do to prevent those threats in milliseconds.

Attackers Have Adapted To Ephemeral Containers

A striking revelation from the Sysdig report is the increasingly transient life of containers. Approximately 70% of containers now have a lifespan of less than five minutes. While this ephemeral nature can be beneficial for resource management, it also presents unique security challenges. Attackers, adapting to these fleeting windows, have honed their methods to conduct swift, automated reconnaissance. The report highlights that a typical cloud attack unfolds within a mere 10 minutes, underscoring the need for real-time response actions.

How To Prevent Data Exfiltration in Ephemeral Containers

Many organizations have opted to use open-source Falco for real-time threat detection in cloud-native environments. In cases where the adversary opts to use an existing tool such as kubectl cp to copy artifacts from a container’s file system to a remote location via the Kubernetes control plane, Falco can trigger a detection within milliseconds.

- rule: Exfiltrating Artifacts via Kubernetes Control Plane
  desc: Detect artifacts exfiltration from a container's file system using kubectl cp.
  condition: >
    open_read 
    and container 
    and proc.name=tar 
    and container_entrypoint 
    and proc.tty=0 
    and not system_level_side_effect_artifacts_kubectl_cp
  output: Exfiltrating Artifacts via Kubernetes Control Plane (file=%fd.name evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty)
  priority: NOTICE
  tags: [maturity_incubating, container, filesystem, mitre_exfiltration, TA0010]

This Falco rule can identify potential exfiltration of application secrets from ephemeral containers’ file systems, potentially revealing the outcomes of unauthorized access and control plane misuse via stolen identities (such as stolen credentials like Kubernetes serviceaccount tokens). In cases where an attack can start and complete its goal in less than 5 minutes, the need for a quick response action is critical. Unfortunately, this Falco rule alone will only notify users of the exfiltration attempt. We need an additional add-on to stop this action entirely.

Preventing Data Exfiltration With Falco Talon

Falco Talon was recently designed as an open-source response engine for isolating threats, specifically in the container orchestration platform, Kubernetes. It enhances the cloud-detection detection engine Falco with a no-code solution. In this case, developer operations and security teams can seamlessly author simple Talon rules that respond to existing Falco in real-time. Notice how the Talon rule below gracefully terminates a workload if it is flagged as triggering the aforementioned “Exfiltrating Artifacts via Kubernetes Control Plane” Falco rule.

- name: Prevent control plane exfiltration
  match:
    rules:
      - "Exfiltrating Artifacts via Kubernetes Control Plane"
  action:
    name: kubernetes:terminate
    parameters:
      ignoreDaemonsets: true
      ignoreStatefulsets: true
      grace_period_seconds: 0Code language: JavaScript (javascript)

In the example above, the action chooses to utilize the existing Kubernetes primitives for graceful termination with the name “kubernetes:terminate“. It’s important that your application handles termination gracefully so that there is minimal impact on the end-user and the time-to-recovery is as fast as possible – unlike SIGKILL, which is much more forceful.

In practice, this terminate action means your pod will handle the SIGTERM message and begin shutting down when it receives the message. This involves saving state, closing down network connections, and finishing any work that is left.

In Falco Talon, the parameters “grace_period_seconds” specifies the duration in seconds before the pod should be deleted. The value zero indicates delete immediately. If configured, the attacker is instantly kicked out of the session and therefore unable to exfiltrate data.

The Threat of Quick and Agile Attackers

The agility of attackers in the cloud environment cannot be underestimated. Once they gain access, they rapidly acquire an understanding of the environment, poised to advance their malicious objectives. This rapid adaptation means that even short-lived, vulnerable workloads can expose organizations to significant risks. The traditional security models, which rely on longer response times, are proving inadequate against these fast-paced threats.

Conclusion

The insights from the Sysdig report unequivocally call for a strategic reevaluation of security approaches in Kubernetes environments. In response to the challenges posed by limited visibility and the need for effective security controls in ephemeral containers and workloads, projects like the Cloud Native Computing Foundation’s (CNCF) Falco, and its latest open-source companion Falco Talon, have emerged as vital tools. Designed to tackle the intricacies of short-lived (less than 5 minutes) containers, these solutions offer real-time security monitoring and continuous scanning, transitioning from recommended practices to essential components in a Kubernetes security arsenal.

Organizations must find a balance between leveraging the convenience of cloud-native technologies and enforcing stringent security protocols. As attackers increasingly exploit the ephemeral nature of containers, the organizational response must be both dynamic and proactive. Tools like Falco and Falco Talon exemplify the kind of responsive, advanced security measures necessary to navigate this landscape. They provide the much-needed visibility and control to detect and respond to threats in real time, thereby enhancing the security posture in these fast-paced environments.

Ensuring robust cybersecurity in the face of sophisticated threats is undoubtedly challenging, but with the right tools and strategies, it is within reach. The integration of solutions like Falco and Falco Talon into Kubernetes environments is key to safeguarding against today’s advanced threats, ensuring a secure, efficient, and resilient cloud-native ecosystem for tomorrow.