NetSPI Introduces EchoBench, a Human-Calibrated Benchmark for Autonomous Pentesting
Summary
NetSPI introduced EchoBench, a benchmark comparing autonomous web application pentesting systems with associate human testers across finding fidelity, difficulty reach, OWASP breadth, and repeatability.
Key points
- EchoBench evaluates a model together with its harness and compares its results with human testers on the same applications and finding criteria.
- Its ECHO score combines finding fidelity, difficulty reach, OWASP breadth, and repeatability using geometric means; a score of 100 represents parity with the measured human cohort.
- The current corpus covers five web applications and 184 canonical security findings; 173 findings have human-assigned difficulty labels.
- The human reference draws on NetSPI University associate assessments spanning five years and roughly 300 assessments; the current matched snapshot includes nearly 100.
- The first numeric release is pending completion of finding alignment, difficulty labels, and balanced repeated-run coverage.
Article Details
- Event Type
- Introduction of EchoBench, a benchmark for autonomous web application penetration testing
- Impact
- EchoBench establishes a measured associate-pentester reference for evaluating autonomous systems on five web applications and 184 canonical findings. No autonomous-system scores or security-incident impacts are reported.