How Model Extraction Threatens Proprietary AI Systems—and How to Limit It
Summary
The explainer describes API-based model extraction, including Anthropic’s allegation that DeepSeek, Moonshot AI and MiniMax used 24,000 accounts for 16 million Claude exchanges, outlines technical limits, and reviews defenses.
Key points
- Anthropic alleged that DeepSeek, Moonshot AI and MiniMax used roughly 24,000 fraudulent accounts to generate more than 16 million exchanges with Claude, collecting responses to train or improve their models.
- Model extraction uses queries and a model’s responses as training data to imitate its behavior; API access may be sufficient, without access to the original system’s code or infrastructure.
- Confidence scores and detailed outputs can make extraction easier, but researchers demonstrated that predictions alone can also be used to train substitutes.
- A 2024 study recovered the embedding projection layers of OpenAI’s Ada and Babbage models through API access; it did not recover the complete models, and the researchers had OpenAI’s approval.
- Imitating responses does not necessarily reproduce a model’s underlying capabilities. Anthropic’s report describes campaign scale and targets but does not establish how much capability the labs obtained.
- Suggested defenses include rate limits, monitoring patterns across linked accounts, restricting detailed outputs, identity checks and watermarking; each addresses different risks and has limitations.
Article Details
- Topic
- Model extraction through repeated API queries and the risks to proprietary AI systems
People
CarliniResearcher whose work recovered the embedding projection layer of OpenAI’s Ada and Babbage models through API access and examined the limits and cost of parameter extraction.Garry TanArgued against government intervention in model distillation, while also stating that he did not support the use of stolen credentials.GudibandeResearcher whose study found that models trained to imitate ChatGPT could sound convincing but struggled outside their training examples and did not match its factual accuracy.TramèrResearcher whose work demonstrated how confidence values returned by prediction APIs can help attackers reproduce machine-learning models.
Threat Actors
DeepSeekAnthropic alleged that DeepSeek used fraudulent accounts to collect Claude responses for model training. OpenAI separately said it identified accounts linked to DeepSeek employees attempting to bypass access restrictions through obfuscated third-party routers.MiniMaxAnthropic alleged that MiniMax used fraudulent accounts to collect Claude responses for model training, focusing on agentic coding and tool orchestration.Moonshot AIAnthropic alleged that Moonshot AI used fraudulent accounts to collect Claude responses for model training, targeting agentic reasoning, tool use, coding, and computer-use capabilities.
Vendors
AnthropicIn February 2026, Anthropic alleged that three AI labs, DeepSeek, Moonshot AI, and MiniMax, had used roughly 24,000 fraudulent accounts to generate more than 16 million exchanges with Claude.GoogleThe campaigns described by Anthropic and Google follow this approach. They involve collecting responses from competing models without authorization.OpenAIResearchers demonstrated this in a paper presented at ICML 2024. Carlini and colleagues recovered the embedding projection layer of OpenAI’s Ada and Babbage models through ordinary API access.
Products
AdaResearchers demonstrated this in a paper presented at ICML 2024. Carlini and colleagues recovered the embedding projection layer of OpenAI’s Ada and Babbage models through ordinary API access.BabbageResearchers demonstrated this in a paper presented at ICML 2024. Carlini and colleagues recovered the embedding projection layer of OpenAI’s Ada and Babbage models through ordinary API access.ChatGPTA 2023 study by Gudibande and colleagues found that models trained to imitate ChatGPT could produce responses convincing enough to impress human evaluators. However, they struggled with tasks outside their trainingClaudeIn February 2026, Anthropic alleged that three AI labs, DeepSeek, Moonshot AI, and MiniMax, had used roughly 24,000 fraudulent accounts to generate more than 16 million exchanges with Claude.GeminiGoogle’s report also shows how specific these attempts can get. One campaign instructed Gemini to reveal its full reasoning process. Google said the more than 100,000 prompts suggested an attempt to replicate Gemini’sGPT-3.5 TurboThey also estimated that extracting the full projection matrix of GPT-3.5 Turbo models would cost around $2,000 for one variant and $8,000 for another. They did not attempt either extraction.