Wiz Researchers Detail SAP AI Core Vulnerability Chain That Exposed Customer Secrets
Summary
Wiz researchers say vulnerabilities in SAP AI Core let them bypass network restrictions, access internal services and customer cloud credentials, and take over the Kubernetes cluster. SAP fixed the issues; Wiz says no customer data was compromised.
Key points
- Researchers used legitimate AI training workflows to run code in SAP AI Core, then bypassed network restrictions by sharing a process namespace and running as the Istio user.
- An exposed Grafana Loki configuration revealed AWS credentials that provided access to an S3 bucket containing service and customer-Pod logs.
- Researchers found six unauthenticated EFS shares containing customer-organized AI data, including code and training datasets.
- An unauthenticated Helm Tiller service exposed credentials for SAP’s internal container registry and Artifactory, with access that could enable image or build poisoning.
- Using Tiller’s write access, researchers deployed a Pod with Kubernetes cluster-admin privileges and accessed customer Pods, secrets, and a Google Container Registry key with read/write permissions.
- Wiz reported the vulnerabilities to SAP beginning January 25, 2024; SAP deployed fixes for all reported issues by May 15, 2024. Wiz says no customer data was compromised.
- The researchers highlight tenant-isolation risks in managed AI services and recommend defense in depth, including stronger isolation for untrusted code and hardening internal services.
Article Details
- Attack Vectors
- A customer-configured Argo Workflow spawned a Kubernetes Pod that could run arbitrary code, allowing the researchers to probe for cross-tenant access weaknesses.
- The Pod bypassed network restrictions by sharing its process namespace with the Istio sidecar to obtain an Istio token, then running under Istio's UID (1337), which was excluded from Istio's traffic rules.
- The researchers used the token to inspect Istiod configuration and reach internal services, including a Grafana Loki /config endpoint that exposed AWS credentials and unauthenticated EFS shares containing customer files.
- An unauthenticated Helm Tiller server exposed registry and Artifactory credentials. Its write operations also allowed deployment of a Helm package that created a Pod with cluster-admin privileges.
- Cluster-admin access exposed customer secrets and enabled potential access to customer Pods and data, as well as modification of internal images, builds, and customer workloads.
- Defensive Notes
- SAP fixed the reported vulnerabilities by May 15, 2024, and rotated relevant secrets after its first fix on February 16, 2024.
- Harden admission controls to prevent unsafe Pod configurations, including process-namespace sharing and UID settings that bypass network restrictions.
- Enforce authentication and least privilege on internal services, including Loki, EFS, Helm Tiller, registries, and artifact repositories.
- Separate customer workloads from control-plane and internal assets with robust tenant isolation, network controls, and sandboxing for untrusted AI training code.
- Restrict and rotate cloud and service credentials, and limit their permissions to the minimum required.
MITRE ATT&CK
T1046 · Network Service DiscoveryAfter bypassing the Pod's network restrictions, the researchers scanned its internal network to identify services.T1552.001 · Credentials In FilesThe researchers retrieved AWS credentials from Loki's exposed configuration and customer secrets from Kubernetes.T1609 · Container Administration CommandThe researchers used the Helm Tiller server's install operation to deploy the privileged Pod to the Kubernetes cluster.T1610 · Deploy ContainerThe researchers installed a malicious Helm package that deployed a Pod with cluster-admin privileges.
People
Hillai Ben-SassonWiz Research Team member named as a researcher on the SAP AI Core findings.Nir OhfeldWiz Research Team member named as a researcher on the SAP AI Core findings.Ronen ShustinWiz Research Team member named as a researcher on the SAP AI Core findings.Sagi TzadikWiz Research Team member named as a researcher on the SAP AI Core findings.Shir TamariWiz Research Team member named as a researcher on the SAP AI Core findings.
Vendors
SAPFor the latest installment of this project, we researched SAP’s AI offering, aptly named “SAP AI Core.” This is our 3rd report in the series, following our research on the Hugging Face and Replicate platforms. This blogWizOver the past months, we on the Wiz Research Team have conducted extensive tenant isolation research on multiple AI service providers. We believe these services are more susceptible to tenant isolation vulnerabilities,
Products
Google Container RegistryRead and modify SAP’s Docker images on Google Container Registry Grafana LokiWe found an instance of Grafana Loki on the cluster, so we requested the /config endpoint to view Loki’s configuration. The API responded with the full configuration, including AWS secrets that Loki used to access S3: HelmBug #4: Unauthenticated Helm server compromises internal Docker Registry and Artifactory IstioWe quickly realized our Pod had extremely limited network access, as enforced by an Istio proxy sidecar – so scanning the internal network wasn’t an option for us.JFrog ArtifactoryRead and modify artifacts on SAP’s internal Artifactory server KubernetesGain cluster administrator privileges on SAP AI Core’s Kubernetes cluster SAP AI CoreFor the latest installment of this project, we researched SAP’s AI offering, aptly named “SAP AI Core.” This is our 3rd report in the series, following our research on the Hugging Face and Replicate platforms. This blog