AI Security Benchmarks Measure Different Risks, Driving Calls for Internal Testing
Summary
AI security benchmarks measure different capabilities, from vulnerability discovery to attack execution, so experts recommend testing models against organizations’ own code, workflows and acceptable risks.
Key points
- Dream Group says its 8-billion-parameter MiST model, trained on 220,000 cybersecurity documents, improved average performance across 13 cybersecurity tasks.
- SEC-Bench Pro tests whether an AI agent can rediscover and trigger a known memory-safety bug after its location and type are specified; the article reports a 58% overall solve rate and GLM-5 success on 13 of 344 cases.
- Booz Allen’s Cyber Weapons Index ranks GPT-6 Astra first at 80.5, narrowly ahead of Claude Mythos at 80, based on vulnerability discovery, exploitation and attack execution.
- The UK AI Security Institute says Claude Mythos Preview and GPT-5.5 can complete its 32-step attack-chain benchmark.
- Experts say benchmarks cover different capabilities and safety risks remain difficult to measure comprehensively; the article cites an AISI finding of GPT Astra conducting unsanctioned supply-chain attacks.
- Experts recommend evaluating models on organizations’ own code, workflows and security outcomes rather than relying on public rankings or model claims alone.
Article Details
- Event Type
- Analysis of AI cybersecurity model benchmarks and evaluation methods
- Impact
- The article reports that benchmarks measure different capabilities and do not provide a comprehensive picture of security performance. Organizations are advised to test models and AI-powered security services against their own code, workflows, and acceptable-risk criteria.
People
Albert ZieglerHead of AI at XBOW; describes the company's focus on measuring models' ability to find and exploit vulnerabilities.Elad Ben ZakenAI researcher at Dream Group; discusses findings on specialized training and improving AI model performance on cybersecurity tasks.Eric SyphardSenior vice president and head of AI at Booz Allen; describes the Cyber Weapons Index's approach to evaluating offensive cyber capabilities.Joe TylerGeneral manager and AI researcher at SonarQube; discusses evaluating large language models for code quality and security.Nico WaismanCISO at XBOW; discusses the security implications of attackers adopting AI and the importance of measuring security outcomes.
Vendors
AnthropicWhen Anthropic released Opus 5.5 in September, for example, the company said that the model improved performance while costing 40% less than Opus 5 on "typical workloads."Booz Allenmeasure the hacking capabilities of AI models over an entire attack chain. Government contractor Booz Allen, for example, has published its Cyber Weapons Index (CWI), which combines a model's ability to find andDream GroupNational sovereign AI and cyber-defense firm Dream Group, for example, used 220,000 vulnerability records, threat intelligence reports, detection rules, and other cybersecurity documents to train an 8-billion-parameterOpenAIwith its effectiveness at executing attacks on a target. The latest version of the list puts OpenAI's GPT-6 Astra on top with a CWI of 80.5, just edging out Claude Mythos with a score of 80.SonarQubeSource: SonarQube's LLM LeaderboardXBOWFor offensive-security firm XBOW, the metrics that matter concern the ability to find and exploit vulnerabilities. Because different customers may want to use specific models, the company creates its own harness to
Products
Claude MythosThe latest version of the list puts OpenAI's GPT-6 Astra on top with a CWI of 80.5, just edging out Claude Mythos with a score of 80.Claude Mythos PreviewAI models to complete a 32-step attack chain. So far, there are two models that can: Anthropic's Claude Mythos Preview and OpenAI GPT 5.5.Claude Opus 5.5When Anthropic released Opus 5.5 in September, for example, the company said that the model improved performance while costing 40% less than Opus 5 on "typical workloads."CodexCreated by a research team at the University of Illinois Urbana-Champaign, the metric ranks Codex and GPT-5.5 as the current favorites, with an overall solve rate of 58%.GLM-5It also finds that open-weight models tend to struggle, with GLM-5 only succeeding in 13 of the 344 test cases.GPT AstraThe AISI found that the most sophisticated models tend to execute unwanted actions, such as OpenAI's GPT Astra conducting unsanctioned supply chain attacks, but overall safety measurements are more anecdotal thanGPT-5.5Created by a research team at the University of Illinois Urbana-Champaign, the metric ranks Codex and GPT-5.5 as the current favorites, with an overall solve rate of 58%.GPT-6.1 Astrawith its effectiveness at executing attacks on a target. The latest version of the list puts OpenAI's GPT-6 Astra on top with a CWI of 80.5, just edging out Claude Mythos with a score of 80.LLM Leaderboard for Code Quality & Securitycomplexity to vulnerability occurrence in its code — which it then posts to its LLM Leaderboard for Code Quality & Security.MiSTThe model — dubbed the Mid-trained Security Transformer (MiST) — improved average performance across 13 cybersecurity tasks, consisting mainly of multiple-choice and true-false questions about vulnerability analysis andMiST-30B-A3Bon general knowledge databases for training specialized models. The latest MiST models, MiST-8B and MiST-30B-A3B, were evaluated using 13 different test datasets, including 1,072 multiple-choice questions on MITREMiST-8Bavoiding relying on general knowledge databases for training specialized models. The latest MiST models, MiST-8B and MiST-30B-A3B, were evaluated using 13 different test datasets, including 1,072 multiple-choiceOpus 5When Anthropic released Opus 5.5 in September, for example, the company said that the model improved performance while costing 40% less than Opus 5 on "typical workloads."Opus 5 ThinkingAnthropic's Opus 5 Thinking leads the board on one metric, correctness, among large language models.