Kubernetes API Visibility Gaps Can Hide Non-Standard Pods and Containers

Summary

Kubernetes API queries may miss some static and pause containers. The article explains how to identify other non-standard pods and containers and how attackers could abuse these visibility gaps for stealth or persistence.

Key points

Static pods are managed by the kubelet, not the Kubernetes API; mirror pods can make them visible when kubelet reporting is configured.
Attackers with access to a static pod manifest location could create privileged pods or alter existing static pods for stealthy persistence.
Init containers appear under spec.initContainers and should be monitored because they can affect application setup.
Pause containers are not reported through the Kubernetes API; identifying them requires node-level access and querying the container runtime.
The article demonstrates how replacing the kubelet’s pause-container image could cause an attacker-controlled container to run with each new pod on that node.
Ephemeral containers can be attached to running pods for debugging and may provide access to pod data and secrets; they appear under ephemeralContainers.
The article recommends supplementing Kubernetes API monitoring with workload runtime agents and checking which users can attach ephemeral containers in production.

Article Details

Defense Focus: Improve visibility into Kubernetes pods and containers that the Kubernetes API may omit, and detect anomalous use of them for stealth or persistence.
Detection Methods: Inspect kubelet command-line arguments and configuration for static pod manifest locations and reporting settings.
Identify mirror pods through their ownerReferences.kind attribute and distinguish them from other pods.
Review pod specifications for initContainers and ephemeralContainers, and compare them with expected deployment contents.
Use node-level access to query the container runtime for pause containers that are not visible through Kubernetes clients.
Check the configured pause-container image, its source, and whether it has changed.
Data Sources: Kubernetes API workload and pod objects
Pod specification fields, including containers, initContainers, and ephemeralContainers
Pod status
Kubelet command line and configuration file
Node-level container runtime inventory
Rule Types: kubectl queries using jq filters over Kubernetes JSON output
kubectl JSONPath queries
Platforms: Kubernetes clusters
Kubernetes nodes and their container runtimes
Defensive Actions: Do not rely solely on the Kubernetes API for complete pod and container visibility; supplement it with workload runtime agents.
Monitor static pod manifest locations and determine whether kubelets report static pods through mirror pods.
Review static pods for unexpected privileged settings or added containers, including changes to existing pods such as kube-proxy.
Investigate unexpected pause-container images or changes to the configured image.
Monitor init containers for unexpected setup activity.
Restrict and review who can attach ephemeral containers to production pods.