Wiz researchers demonstrate cross-tenant attack paths in Hugging Face AI services

Summary
Wiz researchers report using a malicious pickle-based model and container-build weaknesses to demonstrate code execution and cross-tenant risks in Hugging Face services. Hugging Face collaborated on the findings and added security scanning.
Key points
- A crafted pickle-based model uploaded to Hugging Face’s Inference API executed code when loaded, giving researchers a reverse shell.
- Researchers describe using access to Amazon EKS node metadata to obtain a node identity, inspect Kubernetes secrets, and demonstrate potential lateral movement.
- In Hugging Face Spaces, researchers report that insufficiently scoped access to an internal container registry allowed them to pull and overwrite images, creating potential supply-chain risk.
- Wiz recommended restricting pod access to instance metadata with IMDSv2 and a hop limit, and enforcing authentication and limited access for internal container registries.
- Hugging Face collaborated with Wiz; the article says it implemented Wiz CSPM and vulnerability scanning and was undergoing its annual penetration test.
Article Details
- Attack Vectors
- Wiz Research uploaded a specially crafted PyTorch Pickle model that executed code when loaded through Hugging Face Inference API, yielding a reverse shell and shell-like command execution.
- Wiz Research reported that it could query the Amazon EKS node’s IMDS from its pod, obtain the node identity, and use aws eks get-token to generate a Kubernetes token with the node role.
- Using the Kubernetes token, Wiz Research listed pods and retrieved secrets associated with its pod; the article says these secrets could enable lateral movement within the EKS cluster.
- In Hugging Face Spaces, a researcher-supplied Dockerfile executed code during the image build. The build environment could push to and pull from a shared internal container registry, allowing images belonging to other customers to be overwritten.
- The article identifies malicious AI applications taking over shared CI/CD infrastructure as a potential supply-chain attack scenario; it does not report that this scenario was carried out.
- Defensive Notes
- Enable IMDSv2 with a Hop Limit that prevents pods from accessing the instance metadata service and obtaining a node role.
- Require authentication for internal container registries and restrict access to the minimum necessary scope.
- Run untrusted AI models in a sandboxed environment, and assess model integrity before using models from untrusted sources.
- Maintain visibility and governance across the AI stack and assess risks including malicious models, training-data exposure, sensitive training data, and AI SDK vulnerabilities.
MITRE ATT&CK
T1059.006 · PythonA crafted Pickle-based model executed code when loaded; Wiz Research also hooked Python functions to create shell-like functionality.T1078.004 · Cloud AccountsWiz Research used the node role obtained through IMDS with aws eks get-token to generate a Kubernetes token.T1195.002 · Compromise Software Supply ChainWiz Research reported write access to a shared internal container registry, which could allow customer images to be overwritten and enable supply-chain attacks; it does not report actually overwriting images.T1552.005 · Cloud Instance Metadata APIFrom its pod, Wiz Research queried the node’s IMDS to obtain the node identity and role.T1552.007 · Container APIWiz Research used kubectl to retrieve secrets associated with its pod; the article says those secrets could enable lateral movement.T1611 · Escape to HostWiz Research says it used container-escape techniques to break out of tenant isolation; the article does not detail the escape mechanism.T1613 · Container and Resource DiscoveryAfter generating the Kubernetes token, Wiz Research listed pods in the EKS cluster.
People
Hillai Ben-SassonNamed as a member of the Wiz Research team behind the research.Nir OhfeldNamed as a member of the Wiz Research team behind the research.Ronen ShustinNamed as a member of the Wiz Research team behind the research.Sagi TzadikNamed as a member of the Wiz Research team behind the research.Shir TamariNamed as a member of the Wiz Research team behind the research.
Vendors
Amazon Web ServicesFor ease of understanding, this can be compared, at a high level, to consuming cloud infrastructure from AWS/GCP/Azure to run your applications and code. Wiz Research was able to compromise the service running theHugging FaceIn this blog we outline our joint work with Hugging Face, one of the best-known AI-as-a-Service providers. Hugging Face has undergone a meteoric rise and grown at an unprecedented rate to meet swelling demand. What weWizWhen we move fast, we break things. In recent months, Wiz Research partnered with AI-as-a-Service companies to uncover common security risks that may impact the industry and subsequently put users’ data and models at
Products
Amazon Elastic Kubernetes ServiceAmazon EKS privilege escalation via IMDS Hugging Face Inference APIrun, which is often outsourced to AI service providers. In Hugging Face, this service is called Hugging Face Inference API. For ease of understanding, this can be compared, at a high level, to consuming cloudInference EndpointsInference Infrastructure: The infrastructure that allows execution of the AI model — being “on edge” (like Transformers.js) or via API or Inference-as-a-Service (like Hugging Face’s Inference Endpoints).PyTorchHugging Face’s platform supports various AI model formats. By performing a quick search on Hugging Face, we can see that two formats are more prominent than others: PyTorch (Pickle) and Safetensors. SpacesSpaces – which offers a simple way to host AI/ML applications, for the purpose of showcasing AI models or working collaboratively on developing AI-powered applications.Wiz CSPMHugging Face has recently implemented Wiz CSPM and vulnerability scanning to proactively identify and mitigate some of the toxic risk combinations found here. In addition, Hugging Face is also currently going through
Tools
aws eks get-tokenobtain its identity, we could also obtain the role of a Node inside the EKS cluster by abusing the aws eks get-token command. This is a fairly common misconfiguration (/ insecure default) in Amazon EKS. It is popularkubectlOne of the things we did was to list information about the Pod where we were running via kubectl get pods/$(hostname), and then view the secrets that are associated with our pod. We were able to prove that by obtainingnetstatAfter executing code in the building process of our image, we used the netstat command to examine network connections made from our machine. One connection was to an internal container registry to which our built layersPickle Scanningused to examine the integrity of a given model and verify that it is indeed not malicious (such as Pickle Scanning by Hugging Face) — so developers and engineers must be very careful deciding where to download the