Aikido Benchmark Reports GPT-5.6 and Kimi K3 Rediscovered 23 of 26 CVEs

Summary

Aikido reports that GPT-5.6 and Kimi K3 each rediscovered 23 of 26 known CVEs in its code-analysis benchmark. Repeated runs of cheaper models often found more bugs than a single run of a pricier model.

Key points

Aikido tested AI models against 26 known vulnerabilities from the GitHub advisory database, spanning different languages and project types.
The benchmark reports GPT-5.6 reached 23/26 findings; a later test gave Kimi K3 the same score.
Models were run three times, with findings pooled as pass@3; results varied between runs, and pooling often increased the number of CVEs found.
For testing, agents were pointed directly to vulnerable code snippets, so misses were intended to reflect reasoning rather than repository navigation.
All tested models found the two critical vulnerabilities, while an indirect SQL injection was found only by GPT-5.5 and the strongest GPT-5.6 models.
Aikido reports that three runs of lower-cost models could match or exceed the findings of a single flagship-model run for less cost.

Article Details

Publisher: aikido.dev
Scope: Benchmark of AI models rediscovering known vulnerabilities in repositories using an AI Code Analysis harness.
Sample Size: 26 known vulnerabilities; 13 models in the original benchmark, with Kimi K3 added in an update.
Key Statistics: GPT-5.6 reached 23/26 vulnerabilities (88.5%) in the reported pass@3 results.
Kimi K3 rediscovered 23/26 vulnerabilities in the updated benchmark.
GLM-5.2 rediscovered 16/26 vulnerabilities (59%) in the reported results.
Three runs of gpt-5.4-nano pooled to 18/26, while no single run found more than 13.
Every tested model rediscovered both critical vulnerabilities in the dataset.
Recommendations: Consider running a model multiple times and pooling findings; the report found this could outperform a single run of a more expensive model.
Use a code-analysis harness that directs investigation toward relevant code and triages reported findings.