Mid-Tier AI Models Show Gains in Offensive Security Testing
Summary
XBOW testing found mid-tier models increasingly capable in offensive security, with GPT-5.5 improving web application testing. Anthropic reported coordinated agents found 266 vulnerabilities across 15 open-source projects, versus 21 for individually assigned agents.
Key points
- XBOW reported that GPT-5.5 improved on GPT-5 in web application testing, with a 10% vulnerability miss rate compared with GPT-5’s 40%.
- In XBOW’s tests, live interaction with a running system contributed more to findings than access to its source code in some comparisons.
- Anthropic reported that a coordinating agent swarm found 266 vulnerabilities across 15 open-source projects, compared with 21 found by agents working individually.
- The two Anthropic agent tests used 27 million and 6.5 million tokens, respectively.
- XBOW found Mythos Preview strong at vulnerability discovery and source-code reasoning, but less effective at exploiting vulnerabilities; losing live-site access also reduced performance.
- The article warns that lower model costs could make repeated offensive-security tasks more accessible, while noting that responsible use still requires human management.
Article Details
- Event Type
- Research findings on AI models' offensive cybersecurity capabilities
- Impact
- XBOW testing reported that lower-cost, increasingly capable models can perform strongly on vulnerability discovery and exploitation tasks, especially when given more time and live-system access. In a separate Anthropic experiment, a coordinating agent swarm found 266 vulnerabilities across 15 open-source projects, compared with 21 found by agents working individually. The article warns that lower costs and repeatable use could make these capabilities more accessible to malicious hackers; it reports no specific attack or victim.
People
Vendors
Anthropic#GPT5_5Metaand vulnerability-finding performance from models including GPT 5.5, GLM-5.2, Grok 4.5, Opus 4.7, and Meta’s Muse Spark 1.1, with agent swarms and live-system interaction proving especially powerful.OpenAI#GPT5_5xAI#GPT5_5Z.aiModels like Z.ai’s open-weight GLM-5.2, xAI’s Grok 4.5, Anthropic’s Opus 4.7, Meta’s Muse Spark 1.1, still perform very strongly at many hacking and exploitation tasks that worry policymakers.
Products
Claude Mythos PreviewNew research this week from Anthropic tested two models – Mythos Preview, which is used in Project Glasswing, and Opus 4.8 – to learn how quickly multi-agent swarms could find vulnerabilities in 15 open-source softwareGLM-5.2testing showed strong exploitation and vulnerability-finding performance from models including GPT 5.5, GLM-5.2, Grok 4.5, Opus 4.7, and Meta’s Muse Spark 1.1, with agent swarms and live-system interaction provingGPT 5New testing showed strong exploitation and vulnerability-finding performance from models including GPT 5.5, GLM-5.2, Grok 4.5, Opus 4.7, and Meta’s Muse Spark 1.1, with agent swarms and live-system interaction provingGPT 5.6Frontier models like Mythos and GPT 5.6 are indeed more capable on individual cybersecurity tasks, but they can also come with exponentially higher token costs.GPT-5.5New testing showed strong exploitation and vulnerability-finding performance from models including GPT 5.5, GLM-5.2, Grok 4.5, Opus 4.7, and Meta’s Muse Spark 1.1, with agent swarms and live-system interaction provingGrok 4.5showed strong exploitation and vulnerability-finding performance from models including GPT 5.5, GLM-5.2, Grok 4.5, Opus 4.7, and Meta’s Muse Spark 1.1, with agent swarms and live-system interaction proving especiallyMuse Spark 1.1performance from models including GPT 5.5, GLM-5.2, Grok 4.5, Opus 4.7, and Meta’s Muse Spark 1.1, with agent swarms and live-system interaction proving especially powerful.Mythos#GPT5_5Opus 4.7exploitation and vulnerability-finding performance from models including GPT 5.5, GLM-5.2, Grok 4.5, Opus 4.7, and Meta’s Muse Spark 1.1, with agent swarms and live-system interaction proving especially powerful.Opus 4.8this week from Anthropic tested two models – Mythos Preview, which is used in Project Glasswing, and Opus 4.8 – to learn how quickly multi-agent swarms could find vulnerabilities in 15 open-source software projects