Wiz Researchers Detail Cross-Tenant Vulnerability in Replicate AI Platform
Summary
Wiz researchers used a malicious Cog container and TCP injection into an authenticated Redis connection to access and alter other customers’ prompts and outputs. Replicate fixed the issue after disclosure in January 2024; no customer data was compromised.
Key points
- Researchers uploaded a malicious Cog container and used it to achieve root-level code execution inside their container on Replicate’s infrastructure.
- The container shared a network namespace with a neighboring container, exposing an established plaintext connection to a Redis server.
- Using TCP injection, researchers sent commands over the neighboring container’s authenticated Redis connection and found the queue served multiple customers.
- A Lua script injected into Redis changed a queued prediction’s webhook, allowing researchers to receive its prompt and output and alter the output.
- The flaw could have exposed customer prompts and private-model outputs or enabled their manipulation; the researchers said exploitation could have put sensitive data at risk.
- Wiz disclosed the issue in January 2024. Replicate promptly mitigated it, and Wiz said no customer data was compromised and customers need take no action.
Article Details
- Attack Vectors
- Uploaded a malicious Cog container to Replicate and interacted with it to achieve remote code execution as root inside its worker pod.
- Used tcpdump to inspect an established plaintext Redis connection shared through the pod’s network namespace, then used rshijack to inject commands into the authenticated TCP session.
- Injected a Lua script into Redis to redirect another queue item’s webhook to a rogue API server, which received the prediction input and output and could modify the output.
- Defensive Notes
- Replicate promptly implemented mitigations after responsible disclosure in January 2024.
- The researchers reported that no customer data was compromised and that customers did not need to take action.
MITRE ATT&CK
T1040 · Network SniffingAfter gaining root access and network capabilities in the pod, the researchers used tcpdump to inspect traffic on an established Redis connection.T1565.002 · Transmitted Data ManipulationThe researchers injected a Lua script into the Redis connection to alter a queued prediction item’s webhook and redirect its data to their rogue API server.T1610 · Deploy ContainerThe researchers uploaded a malicious Cog container to Replicate and ran it on the platform to obtain code execution in a worker pod.
Vendors
Products
Coginference of AI models on their platform, Replicate uses their own format for containerizing models named Cog. Cog helps users containerize their AI models, include the correct dependencies and libraries, and bundle aGoogle Cloud PlatformWe quickly noticed that we were running within a pod inside a Kubernetes cluster hosted on Google Cloud Platform. The pod we were running in was not a privileged pod, so we could not easily escape to the node.Kuberneteswe started investigating the environment. We quickly noticed that we were running within a pod inside a Kubernetes cluster hosted on Google Cloud Platform. The pod we were running in was not a privileged pod, so weRedisthe contents of this TCP connection. Looking at content of the stream we realized it was a plaintext Redis protocol (a popular, open-source, in-memory data store). By performing a reverse DNS lookup on the IPReplicatewith the Hugging Face team. Now, in this second installment, we will detail a vulnerability in the Replicate AI platform.
Tools
netstatWe started investigating the network. By using netstat we noticed that there was an already-established TCP connection handled by a process that was running in a different PID namespace. This made us conclude that weredis-cliWe tried to initiate a new connection to this Redis server using the redis-cli, but the server required credentials for authentication.rshijackWe used rshijack (a utility for TCP injection) to inject arbitrary packets to the existing TCP connection to the Redis server of our neighboring container in order to bypass the authentication. It worked! We performedtcpdumpAs we were running with root privileges and were capable of CAP_NET_RAW and CAP_NET_ADMIN, we could use tcpdump to examine the contents of this TCP connection. Looking at content of the stream we realized it was a