Kubernetes Audit Log Gaps Can Hinder Detection and Forensics

· Original article ↗

Summary

The article explains how inconsistent Kubernetes audit logging across cloud and self-hosted clusters can leave detection and forensic gaps, and recommends centralizing logs, normalizing formats, and adding other data sources.

Key points

  • Audit logs provide Kubernetes-specific details about who performed an action, what resource was affected, and when it happened—context that general cloud or network logs may not provide.
  • The article says audit logging is disabled by default in Amazon EKS and Azure AKS, while Oracle Kubernetes Engine enables it by default but does not let users disable it.
  • Cloud providers may change Kubernetes audit-log formats, causing detection rules written for one format to miss events in another and requiring separate rulesets or log normalization.
  • Users may have limited visibility into provider-controlled audit policies, making it difficult to know which events are logged.
  • Log streaming delays can hinder real-time response, while storing and querying large volumes of logs can be costly.
  • The article recommends consistently configuring and centralizing audit logs, normalizing formats where possible, and supplementing them with admission-controller, cloud, and network logs.

Article Details

Defense Focus
Improve Kubernetes threat detection and forensic coverage by consistently enabling and centralizing audit logs, accounting for provider-specific formats and policy gaps, and augmenting audit data with other telemetry.
Detection Methods
  • Configure Kubernetes audit logging consistently across clusters and send events to a centralized SIEM.
  • Use detection rules for suspicious Kubernetes API activity, including workload creation by anonymous or unauthenticated users; adapt rules to provider-specific log formats or normalize events first.
  • Correlate audit events with dynamic admission-controller state-change monitoring, cloud-specific logs, and network telemetry.
  • Use audit events to investigate the actor, source IP, action, resource, and event timestamp.
Data Sources
  • Kubernetes audit logs
  • Dynamic admission-controller events
  • AWS CloudTrail
  • Azure Activity Logs
  • VPC Flow Logs
  • Network Policies
  • Network behavior monitoring
Rule Types
  • Rego detection rules for Kubernetes workload creation by anonymous or unauthenticated users
Platforms
  • SIEM
Defensive Actions
  • Verify audit logging is enabled across all managed and self-hosted clusters; some providers disable it by default.
  • Assess provider-specific audit policy coverage and identify events or fields that may be omitted.
  • Normalize provider-specific audit events where possible, or maintain rules compatible with each format.
  • Account for ingestion latency and storage and query costs when centralizing high-volume logs.
  • Augment audit logs with admission-controller, cloud infrastructure, and network telemetry to improve detection and investigations.

Vendors

Products

Amazon Elastic Kubernetes ServiceCSPs is the inconsistency in how Kubernetes audit logging is configured by default. For example, in Amazon EKS (Elastic Kubernetes Service) and Azure AKS (Azure Kubernetes Service), audit logging is disabled byAzure Kubernetes Serviceaudit logging is configured by default. For example, in Amazon EKS (Elastic Kubernetes Service) and Azure AKS (Azure Kubernetes Service), audit logging is disabled by default. Administrators must manually enableAzure Monitorand performance issues. When audit logs are streamed to centralized systems like CloudWatch or Azure Monitor, there can be significant delays between the time an event occurs and when it appears in the logCloudWatchalso introduce latency and performance issues. When audit logs are streamed to centralized systems like CloudWatch or Azure Monitor, there can be significant delays between the time an event occurs and when it appearsGoogle Kubernetes Enginefor audit logs, many cloud vendors modify or extend this format for their managed services. Google Kubernetes Engine (GKE) and OKE are prime examples: they heavily customize the audit logs, stripping some of theOracle’s Kubernetes EngineContrast this with Oracle’s Kubernetes Engine, where audit logs are turned on by default, but users have no option to disable them. While this ensures that logs are always available, lack of control can introduce issuesWiz Kubernetes security stackGiven the challenges in relying solely on Kubernetes audit logs, the Wiz Kubernetes security stack offers a more comprehensive solution. Wiz combines agentless scanning with real-time detection, using a combination of

Related Articles