Checklist for Evaluating AI Penetration-Testing Vendors
Summary
Aikido outlines criteria for assessing AI pentesting vendors, including live validation, testing depth, code access, reliability, scope controls and workflow fit. It recommends comparing vendors through monitored, comparable test runs.
Key points
- The checklist prioritizes reproducible findings validated against a live system, rather than issues inferred from model output alone.
- It recommends assessing whether agents can test workflows, roles, application state and multi-step attack paths.
- It says code access can improve coverage; Aikido reports that tests with code access found a median of seven times more high- and critical-severity vulnerabilities across more than 1,000 tests.
- Scope and safety criteria include system-enforced restrictions, limiting access to defined domains, and pre-flight checks.
- The article recommends requesting direct access, observing tests live, and comparing vendors using similar scope, setup effort, budget and testing capacity.
- Aikido cites a financial-services example and a Tyro proof of concept to compare AI-assisted testing with manual pentests; these are company-provided examples.
Article Details
- Topic
- Evaluation criteria for AI penetration testing vendors, including validation quality, testing depth, safety controls, reliability, and workflow fit.