1Password Study Finds AI Models Fully Patch Vulnerabilities Less Than Half the Time
Summary
1Password found ChatGPT 5.5 and Claude Opus 4.8 fully patched six high-impact vulnerabilities without introducing bugs in 47% of tests. The article says AI-generated patches need human review.
Key points
- 1Password tested ChatGPT 5.5 and Claude Opus 4.8 on six high-impact vulnerabilities, including the Linux kernel flaw known as Copy Fail.
- The models fully remediated vulnerabilities without erroneous behavior changes in 47% of tests.
- Researchers found patches could address only some vulnerable code paths, add fragile guard code, or introduce subtle behavior changes.
- The article cites Veracode research that found AI models introduced a detectable OWASP Top 10 vulnerability in 44% of its tests.
- Veracode's findings also showed variation across 100 models, with newer models performing better on security tests than many others.
- A Veracode executive said AI-generated patches should be reviewed by knowledgeable humans before acceptance.
Article Details
- Event Type
- Research findings on the reliability of AI-generated security patches
- Impact
- 1Password reported that ChatGPT 5.5 and Claude Opus 4.8 fully patched tested vulnerabilities without introducing new problems in 47% of cases. The research found that patches could leave exploit paths open or introduce bugs, supporting the recommendation that humans review AI-generated patches. Veracode separately reported security vulnerabilities in 44% of its tests of AI-generated code.
People
Axel MierczukNamed as an author of the 1Password research evaluating AI-generated vulnerability patches.John HoodletNamed as an author of the 1Password research evaluating AI-generated vulnerability patches.Spencer MichaelsNamed as an author of the 1Password research evaluating AI-generated vulnerability patches.Tim JarretIdentified as Veracode’s vice president of product; commented on limitations of AI tools for security patching and the need for human review.
Vendors
1PasswordNew research from 1Password found that ChatGPT 5.5 and Claude Opus 4.8 often fail to fully patch high-impact vulnerabilities and can introduce new bugs instead.AnthropicThe findings suggest AI-assisted patching still needs human review, even as models like GPT 5.5, GPT-5.6-Sol, and Anthropic’s Mythos are promoted for stronger security capabilities.OpenAIBut new research that tested the patching capabilities of two popular commercial models, OpenAI’s ChatGPT 5.5 and Anthropic’s Claude Opus 4.8, found that generative AI is more likely to create an exploitable patch orVeracode#ChatGPT
Products
ChatGPT 5.5New research from 1Password found that ChatGPT 5.5 and Claude Opus 4.8 often fail to fully patch high-impact vulnerabilities and can introduce new bugs instead.Claude Opus 4.8New research from 1Password found that ChatGPT 5.5 and Claude Opus 4.8 often fail to fully patch high-impact vulnerabilities and can introduce new bugs instead.GPT-5.5The findings suggest AI-assisted patching still needs human review, even as models like GPT 5.5, GPT-5.6-Sol, and Anthropic’s Mythos are promoted for stronger security capabilities.GPT-5.6 SolThe findings suggest AI-assisted patching still needs human review, even as models like GPT 5.5, GPT-5.6-Sol, and Anthropic’s Mythos are promoted for stronger security capabilities.MythosThe findings suggest AI-assisted patching still needs human review, even as models like GPT 5.5, GPT-5.6-Sol, and Anthropic’s Mythos are promoted for stronger security capabilities.