Security Rules Improved Secure-Code Results for Only One of Three AI Models

Summary

A one-run experiment found that Project CodeGuard rules meaningfully improved GPT 5.5’s security results, but not Sonnet 5’s or Composer 2.5’s. Two rule-enabled builds still had document-access IDOR flaws.

Key points

The author tested Sonnet 5, Composer 2.5, and GPT 5.5 with the same application prompt, with and without Project CodeGuard security rules; each model was run once.
The prompt did not specify document ownership requirements, and the resulting IDOR persisted in the Sonnet 5 and Composer 2.5 builds with security rules.
GPT 5.5 with security rules was the only build to eliminate the cross-user document IDOR; it also added CSRF protections and magic-byte upload validation.
Sonnet 5 with rules retained authorization and session gaps, including read access for deactivated users during the access token’s 15-minute lifetime.
Composer 2.5 with rules retained hardcoded secret fallbacks and tokens that could remain valid for up to 24 hours after logout or password reset.
The author concludes that general security rules did not consistently fix vulnerabilities; explicit requirements in prompts or persistent agent instructions may be needed.

Article Details

Attack Vectors: The generated applications repeatedly had an insecure direct object reference: authenticated users could access or download other users’ documents when document queries were not scoped to the owner.
Composer 2.5 with security rules used hardcoded JWT_SECRET and SESSION_SECRET fallbacks, allowing JWT forgery by anyone who had seen the source code.
Composer 2.5’s tokens could remain usable for up to 24 hours after logout or password reset because revocation was not checked; tokens were stored in localStorage.
File uploads in Composer 2.5 were validated using client-supplied MIME types and file extensions rather than file contents.
The default GPT 5.5 implementation had a CSRF vulnerability in upload handling and trusted client-supplied MIME types.
In the Sonnet 5 implementation with security rules, deactivated or logged-out users retained read access until the 15-minute access token expired because revocation was checked only on mutating requests.
In the GPT 5.5 implementation with security rules, a stolen session cookie could be used from another IP if the attacker also spoofed the same User-Agent string.
Defensive Notes: State access-control requirements explicitly in prompts, including that every document query must be filtered by its owner when documents are meant to be private.
Validate uploaded files using their contents rather than client-controlled MIME types or extensions; the GPT 5.5 security-rules iteration used magic-byte validation.
Ensure revocation is checked in general authentication middleware, not only on CSRF-protected or mutating routes.
Do not use hardcoded fallback values for JWT or session secrets; fail safely when required environment variables are absent.
Use dependency cooldowns; the article recommends a seven-day cooldown and avoiding automatic upgrades when the package manager cannot support one.
Treat security rules as helpful but insufficient on their own; the article reports that results varied by model and that several vulnerabilities persisted.