AI Scales Social Engineering, While Deepfakes Undermine Identity Checks
Summary
AI can make phishing and impersonation faster and more personalized, but layered controls remain effective against many attacks. Deepfakes and biometric injection are harder to detect, so organizations should use independent verification rather than rely on familiar-se,
Key points
- Generative AI helps attackers personalize phishing, research targets, create fake websites, and automate follow-ups, increasing campaign volume while lowering barriers to entry.
- The article cites malicious AI models, legitimate AI website builders, and phishing-as-a-service offerings being used to support social-engineering campaigns.
- Synthetic voices, videos, images, and documents can impersonate trusted people or be inserted directly into identity checks, undermining familiar faces and voices as proof of identity.
- Studies cited in the article found people identified synthetic media with 51.2% accuracy, while a tested deepfake detector performed only slightly better than chance.
- Examples include a 2024 deepfake impersonation that led an Arup employee to transfer about $25 million, and 8,065 reported attempts to bypass one financial institution’s facial-liveness checks between January and August 2025.
- Recommended defenses include phishing-resistant MFA, filtering and account monitoring, verification through a separate trusted channel, and independent approval for sensitive payments and account changes.
Article Details
- Publisher
- Recorded Future
- Scope
- AI-enabled phishing, impersonation, synthetic media, and biometric identity evasion.
- Key Statistics
- In July 2026, LexisNexis Risk Solutions reported a 180% year-on-year increase in attacks involving deepfake documents, images, and videos designed to imitate a live person.
- In February 2024, an Arup employee transferred approximately $25 million after threat actors impersonated executives and employees using deepfake video and synthetic voices.
- Between January and August 2025, Group-IB recorded 8,065 attempts to bypass an unnamed financial institution’s facial-liveness checks.
- A 2024 study found that people identified synthetic images, audio, and video with 51.2% accuracy.
- Following a fraudulent meeting in May 2026, a victim transferred approximately SGD 4.9 million to threat actors.
- Recommendations
- Maintain email and web filtering, require phishing-resistant MFA, and monitor for unusual sign-ins, new inbox rules, unexpected permissions, and suspicious newly hosted websites.
- Revoke active sessions and reset credentials quickly after suspected compromise.
- Train employees to verify unusual requests through a separate trusted channel rather than relying on polished writing, familiar branding, faces, or voices.
- Use deepfake detection alongside independent verification; require a second approver for large payments, payroll changes, access requests, account recovery, and contact-detail changes.
- Pause unusual or urgent requests until a separate check is complete, including when the apparent requester is a senior leader.
MITRE ATT&CK
Threat Actors
Vendors
LovableIn August 2025, Proofpoint reported that threat actors used Lovable, a legitimate AI website builder, to create phishing pages and malware.Z.aiSimilarly, GLM-5.2, a model released by Chinese artificial intelligence company Z.ai (formerly Zhipu AI) in June 2026, was marketed as a tool to support autonomous AI agents in coding and design tasks.
Products
GLM-5.2Similarly, GLM-5.2, a model released by Chinese artificial intelligence company Z.ai (formerly Zhipu AI) in June 2026, was marketed as a tool to support autonomous AI agents in coding and design tasks.LovableIn August 2025, Proofpoint reported that threat actors used Lovable, a legitimate AI website builder, to create phishing pages and malware.
Tools
BlackhatGPTthe proliferation of malicious models such as WormGPT, EscapeGPT, FraudGPT, WolfGPT, DarkGPT, BlackhatGPT, KawaiiGPT, and WormGPT4 allows users to circumvent these controls (1, 2, 3).DarkGPTsocial engineering, the proliferation of malicious models such as WormGPT, EscapeGPT, FraudGPT, WolfGPT, DarkGPT, BlackhatGPT, KawaiiGPT, and WormGPT4 allows users to circumvent these controls (1, 2, 3).EscapeGPTto prevent weaponization for social engineering, the proliferation of malicious models such as WormGPT, EscapeGPT, FraudGPT, WolfGPT, DarkGPT, BlackhatGPT, KawaiiGPT, and WormGPT4 allows users to circumvent theseEvilTokensFor example, EvilTokens, a phishing service that emerged in April 2026, reflects the professionalization of the phishing-as-a-service (PhaaS) ecosystem.FraudGPTweaponization for social engineering, the proliferation of malicious models such as WormGPT, EscapeGPT, FraudGPT, WolfGPT, DarkGPT, BlackhatGPT, KawaiiGPT, and WormGPT4 allows users to circumvent these controls (1, 2,GhostGPTSimilar services such as Nytheon, Xanthorox, GhostGPT, and SheByte follow the same approach, packaging existing tools without safety restrictions within subscription models.KawaiiGPTproliferation of malicious models such as WormGPT, EscapeGPT, FraudGPT, WolfGPT, DarkGPT, BlackhatGPT, KawaiiGPT, and WormGPT4 allows users to circumvent these controls (1, 2, 3).NytheonSimilar services such as Nytheon, Xanthorox, GhostGPT, and SheByte follow the same approach, packaging existing tools without safety restrictions within subscription models.SheByteSimilar services such as Nytheon, Xanthorox, GhostGPT, and SheByte follow the same approach, packaging existing tools without safety restrictions within subscription models.WolfGPTfor social engineering, the proliferation of malicious models such as WormGPT, EscapeGPT, FraudGPT, WolfGPT, DarkGPT, BlackhatGPT, KawaiiGPT, and WormGPT4 allows users to circumvent these controls (1, 2, 3).WormGPTto prevent weaponization for social engineering, the proliferation of malicious models such as WormGPT, EscapeGPT, FraudGPT, WolfGPT, DarkGPT, BlackhatGPT, KawaiiGPT, and WormGPT4 allows users to circumventWormGPT4of malicious models such as WormGPT, EscapeGPT, FraudGPT, WolfGPT, DarkGPT, BlackhatGPT, KawaiiGPT, and WormGPT4 allows users to circumvent these controls (1, 2, 3).XanthoroxSimilar services such as Nytheon, Xanthorox, GhostGPT, and SheByte follow the same approach, packaging existing tools without safety restrictions within subscription models.
Countries
IndiaSimilarly, in April 2026, police in Ahmedabad, Gujarat, India, disrupted an identity-fraud operation that used AI to convert victims’ photographs into synthetic videos of them blinking.SingaporeFurthermore, in May 2026, threat actors employed the same approach to fabricate a Zoom meeting with Singapore’s prime minister, president, government officials, and private-sector representatives.United Kingdomused deepfake video and synthetic voices to impersonate the chief financial officer and other employees of UK-based engineering firm Arup, persuading a legitimate employee to transfer approximately $25 million to theUnited StatesIn December, the US Federal Bureau of Investigation (FBI) also warned that threat actors had used synthetic voice messages to impersonate multiple senior US officials between 2023 and 2025.