AWS Says Continuum Achieved 89% Success on CyberGym-E2E
Summary
AWS reports that Continuum passed 819 of 920 CyberGym-E2E tasks within 90 minutes, finding vulnerabilities, demonstrating them, and producing tested repairs. The benchmark covers memory-safety issues in C and C++ projects.
Key points
- AWS reports Continuum passed 819 of 920 tasks, an 89.0% success rate on CyberGym-E2E’s main end-to-end measure, compared with a previous public high of 65.9%.
- Tasks require an agent to find a vulnerability in a real codebase, demonstrate it with an input, and submit a patch that prevents the failure while preserving tested functionality.
- The 920 tasks cover historical OSS-Fuzz vulnerabilities across 139 open-source projects; agents have 90 minutes and no external network access during execution.
- Continuum uses specialized agents for discovery, validation with proof-of-concept testing, and remediation, then checks that the patch prevents the demonstrated failure and preserves functionality.
- Continuum scored 37.8% on the diagnostic stage requiring repair of the benchmark’s specifically selected vulnerability; AWS says this stage can differ when an agent repairs another valid flaw.
- The benchmark focuses on memory-safety vulnerabilities in C and C++ and does not evaluate other languages, many vulnerability classes, or deployment context.
Article Details
- Event Type
- AWS announcement of autonomous code-security benchmark results
- Impact
- AWS reports that Continuum for code vulnerabilities passed 819 of 920 CyberGym-E2E tasks within the 90-minute limit, achieving an 89.0% end-to-end success rate—23.1 percentage points above the previous public high. The benchmark evaluates vulnerability discovery, proof-of-concept validation, and tested repair in open-source C and C++ projects; it does not assess deployment context and is limited to memory-safety vulnerabilities.