Wiz Details Cyber Model Arena Benchmark and AI Agent Results
Summary
Wiz says its Cyber Model Arena v3.9 tested 59 agent configurations across 327 cybersecurity challenges, finding performance varied by task type and model-harness pairing. The post explains its grading approach, isolation measures and limitations.
Key points
- Wiz reports that Cyber Model Arena v3.9 includes 26 models, four general-purpose harnesses, 59 agents and 327 challenges across code vulnerabilities, API, web and cloud security.
- Average pass@1 varied by task category: 78.0% for API security, 64.6% for web security, 56.5% for cloud security and 38.2% for code vulnerabilities.
- Across 22 models tested with both Claude Code and ADK React, the mean absolute pass@1 difference was 5.1 percentage points, underscoring the effect of the agent harness.
- The benchmark gives agents three attempts per challenge and compares success with solve cost and time.
- Wiz recommends deterministic code graders that check a verifiable effect, plus environment isolation to prevent online answer lookup and grader manipulation.
- Stated limitations include simple harnesses, exact-string grading for code vulnerabilities, possible pretraining contamination from public tasks and confidence intervals not yet published.
Article Details
- Publisher
- Wiz IO
- Scope
- Wiz Cyber Model Arena v3.9 benchmark of AI agents on code vulnerability, API security, web security, and cloud security challenges.
- Sample Size
- 327 challenges; 59 agents; 26 models; four general-purpose harnesses.
- Key Statistics
- Arena v3.9 contains 327 challenges: 221 code vulnerabilities, 54 API security, 29 web security, and 23 cloud security tasks.
- Each challenge allows three attempts and a 1,000-second budget per attempt.
- Across 59 agents, average pass@1 was 78.0% for API security, 64.6% for web security, 56.5% for cloud security, and 38.2% for code vulnerabilities.
- Across 22 models tested with both Claude Code and ADK React, the mean absolute pass@1 gap was 5.1 percentage points.
- Recommendations
- Define the benchmark objective, evaluated subject, task mix, and metrics before creating challenges; use tasks from real distributions where possible.
- Use deterministic code graders to inspect checkable effects rather than relying on an LLM judge.
- Isolate agents from the server and internet and prevent access to answer sources during trials.
- Evaluate the model and harness together, allow multiple attempts, and weigh success against solve cost.
- Start with a small, end-to-end baseline that can be inspected and rerun, then scale after it works.
Products
ADK Reactmodels do better or worse in different harnesses. Across the 22 models we ran on both Claude Code and ADK React, the mean absolute gap was 5.1 pass@1 points. Claude models prefer Claude Code, and Gemini 3.8 FlashAntigravityAntigravity Gemini 4 Argon 77.0 85.1 87.5 0.97 8.4Claude CodeGrok on Claude Code is 9.7% on code and still 71.6% on API.Codexwith GPT-5.6 Sol averages 64.2% pass@1 at $8.29 and 27.2 minutes per solve, however the Pareto frontier Codex + GPT-6 Luna is strictly better, averaging 64.5% pass@1 at only $0.05 and 6.3 minutes per solve.Wiz Cyber Model Arenawe graded, and gathered our tasks is available for download. As of the writing of this blog, the Wiz Cyber Model Arena v3.9 consists of 26 models, four general-purpose harnesses, 59 agents, 327 challenges, three