Testing AI Coding Agents Finds IDOR Vulnerabilities in All Six Generated Apps

Summary
A Datadog Security Labs experiment compared three coding models in plan and default modes. All six generated document portals had IDOR vulnerabilities, and the study found no meaningful correlation between plan mode and more secure code.
Key points
- The experiment tested Sonnet 5, Composer 2.5, and GPT 5.5 in both default and plan modes, using the same application prompt and security review process.
- All six applications had insecure direct object reference vulnerabilities that could let authenticated users access other users’ documents.
- Other findings included exposed password-reset tokens, weak or hardcoded JWT secrets, inadequate upload validation, header injection, and missing rate limits.
- Supply Chain Firewall blocked several vulnerable or outdated dependencies during the builds; some were upgraded to versions the article describes as having no known vulnerabilities.
- Plan mode did not consistently improve security: it introduced severe weaknesses in some implementations while addressing issues found in others.
- The author concludes that explicit security requirements in the prompt had more impact on reducing vulnerabilities than the choice between plan and default mode.
Article Details
- Attack Vectors
- All six generated applications had an IDOR flaw that could let an authenticated user access, download, or comment on documents without verifying ownership.
- One implementation used the hardcoded JWT secret 'dev-secret-change-me' when JWT_SECRET was unset, potentially allowing authentication forgery and administrative impersonation.
- One implementation returned live password-reset tokens in JSON responses when NODE_ENV was not set to 'production'.
- One implementation accepted uploads based only on client-supplied MIME types, allowing MIME-type spoofing.
- Multiple implementations interpolated attacker-controlled filenames into Content-Disposition headers without adequate escaping, enabling header injection.
- Some implementations did not revoke or revalidate tokens after password changes, role changes, account deactivation, or logout.
- One implementation allowed email changes without password re-verification.
- One implementation had a race condition in initial administrative provisioning that could allow concurrent registrations to escalate privileges.
- One implementation bypassed bcrypt for nonexistent accounts, creating a login timing side channel.
- Some implementations lacked rate limiting on authentication or password-reset endpoints, or failed to invalidate earlier reset tokens when issuing new ones.
- One implementation inconsistently applied CSRF protection, leaving state-altering document and user routes exposed.
- Defensive Notes
- Centralize document ownership checks in reusable middleware and apply them to every read, download, and comment route.
- Fail startup if JWT_SECRET is missing or set to a known placeholder; do not silently use insecure defaults.
- Keep password-reset tokens out of response bodies and invalidate prior tokens when issuing new ones.
- Validate uploaded file content with magic-byte sniffing rather than trusting client-supplied MIME types.
- Escape or safely encode filenames before placing them in Content-Disposition headers.
- Revalidate account status and relevant user state against the database, and support token revocation after password or role changes, deactivation, and logout.
- Require password re-verification for sensitive account changes such as email updates.
- Make administrative provisioning atomic or otherwise safe against concurrent registrations.
- Use comparable password-hashing work for nonexistent and existing accounts to reduce timing differences.
- Apply rate limiting consistently to authentication and password-reset endpoints, and enforce CSRF protection on all state-altering routes.
Vendors
Products
ClaudeI used Claude, Cursor, and Codex for their respective models.CodexI used Claude, Cursor, and Codex for their respective models.Composer 2.5For this experiment, I used the same prompt across three different models: Sonnet 5, Composer 2.5, and GPT 5.5.CursorI used Claude, Cursor, and Codex for their respective models.Datadog Code SecurityI used Datadog Code Security to identify software composition analysis (SCA), static application security testing (SAST), and infrastructure-as-code (IaC) vulnerabilities, as well as any code quality suggestions.GPT 5.5For this experiment, I used the same prompt across three different models: Sonnet 5, Composer 2.5, and GPT 5.5.Prisma ORMThis agent delivered the most architecturally mature implementation of the iterations, powered by the Prisma ORM, with dedicated service modules for tokens, mail, and uploads.Sonnet 5For this experiment, I used the same prompt across three different models: Sonnet 5, Composer 2.5, and GPT 5.5.
Tools
Claude's code_review skillTo dig deeper into security vulnerabilities, I used Claude's code_review skill using Sonnet 5 across all iterations to audit the code.Supply Chain Firewall (SCFW)Supply Chain Firewall (SCFW) is an open source command line tool that prevents the installation of malicious npm and Python packages.