Aikido Benchmark Finds DeepSeek V4 Pro Leads AI Vulnerability Discovery
Summary
Aikido benchmarked 10 AI models against 32 recent vulnerabilities, running each three times. DeepSeek V4 Pro found 28 across pooled runs; results also highlight model inconsistency, false leads, and cost trade-offs.
Key points
- The vendor-run benchmark used 32 vulnerabilities from recently disclosed CVEs, three runs per model, and a maximum of 30 agent turns per run; models had no internet access.
- DeepSeek V4 Pro 0813 found 28 of 32 vulnerabilities across three runs, the highest pooled recall, though only 10 appeared in every run.
- Grok 4.6 found 26 vulnerabilities across runs and was the most consistent, finding 21 in all three runs.
- GLM-5.3 found 25 vulnerabilities, including 18 consistently across all runs; Aikido reported it was cheaper than several closed models in its tests.
- Aikido observed that DeepSeek models generated more candidate findings and false leads, increasing the need for downstream triage.
- The article says open-weight models can compete with closed models when properly harnessed, but cautions they are not yet drop-in replacements without careful harness design.
- The benchmark did not include Mythos or GPT 5.6 Daybreak-class models.
Article Details
- Publisher
- Aikido
- Scope
- Benchmark of 10 AI models rediscovering recently disclosed vulnerabilities in real-world repositories.
- Sample Size
- 32 vulnerabilities; each model ran the full set three times (96 runs per model).
- Key Statistics
- The benchmark used 11.7 billion tokens.
- DeepSeek V4 Pro 0813 found 28 of 32 vulnerabilities across three runs, compared with 17 on its first run.
- Grok 4.6 found 21 CVEs in all three runs and 26 across the pooled results, making it the most consistent model tested.
- Kimi K3 had 92.3% pooled precision, the highest among the open-weight models tested.
- 17 cases were found by every model at least once; one case was not found by any of the 10 models.
- Recommendations
- Pool findings across repeated model runs to improve vulnerability recall, and use downstream triage to filter false leads.
Vendors
AikidoWant to see the latest benchmarks? Check out Aikido Arena, which we update regularly.Z.aiThanks to Z.AI, we had the opportunity to work closely with their team as an early evaluation partner for GLM-5.3, putting it through our cybersecurity benchmark and sharing feedback throughout the pre-release process.
Products
AI Code AnalysisThe setup was a bounded version of the harness behind our AI Code Analysis. We replaced the model for the primary stage responsible for code exploration and reasoning with each of our contenders and gave the agent 30Aikido ArenaWant to see the latest benchmarks? Check out Aikido Arena, which we update regularly.Aikido AttackAikido Attack sends autonomous agents at your running app, pooling passes and discarding false leads the way the pipeline here rejected them.ChatGPT: GPT-5.6 Lunafound, going head to head with GPT Sol while being 65.5% cheaper. It also surpassed GPT-5.6 Terra, GPT-5.6 Luna, DeepSeek Flash 0731 and finished just 1 CVE behind Opus 5 while still being 69.8% cheaper.Code Security AuditCode Security Audit runs that reasoning across your whole source.DeepSeek V4 Flash 0731With this, we add GLM-5.3, DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731, Qwen3.8-Max, Kimi K3, and Grok 4.6 to the lineup.DeepSeek V4 Pro 0813With this, we add GLM-5.3, DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731, Qwen3.8-Max, Kimi K3, and Grok 4.6 to the lineup.GLM-5.3With this, we add GLM-5.3, DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731, Qwen3.8-Max, Kimi K3, and Grok 4.6 to the lineup.GPT 5.6 TerraGLM-5.3-preview beat GPT 5.6 Terra on cost-to-performance ratio at pass@3. It rediscovered 75% of the CVEs while costing 40% less and was also the most consistent of the open-weight models with 14/32 found across allGrok 4.6With this, we add GLM-5.3, DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731, Qwen3.8-Max, Kimi K3, and Grok 4.6 to the lineup.Kimi K3With this, we add GLM-5.3, DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731, Qwen3.8-Max, Kimi K3, and Grok 4.6 to the lineup.Opus 5The most expensive model is not required. Three DeepSeek Pro runs cost about $295 and outperform Opus 5, Grok 4.6, or Sol. Three Flash runs cost $108 and reach 24 matching Grok’s best individual pass for less than aQwen3.8-MaxWith this, we add GLM-5.3, DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731, Qwen3.8-Max, Kimi K3, and Grok 4.6 to the lineup.Solmodel is not required. Three DeepSeek Pro runs cost about $295 and outperform Opus 5, Grok 4.6, or Sol. Three Flash runs cost $108 and reach 24 matching Grok’s best individual pass for less than a quarter of