Jason Haddix Demonstrates AI Prompt-Injection Risks in Chatbots
· 18:31 · Watch on YouTube ↗
Summary
At DEF CON, Jason Haddix demonstrates prompt-injection attacks against chatbots, including simulated account-recovery and meeting-summary scenarios. He says the examples reflect reported real-world flaws and highlights repeat testing and free practice labs.
Key points
- Haddix says Arcanum tests AI chatbots to identify unsafe access to other users’ accounts and data.
- The prompt-injection taxonomy organizes testing around attacker goals, attack methods, and evasion, including authority impersonation and encoded or translated prompts to bypass guardrails.
- In a Karen CTF lab, authority impersonation reveals a protected passphrase. Haddix notes that AI responses can vary and says tests may need to be repeated up to 15 times.
- A simulated Instagram account-recovery flow lets an attacker-supplied email receive a verification code and reset another account. Haddix says a bug bounty hunter found a similar flaw at Meta.
- In a recreated meeting app, transcript instructions cause an AI summary to reveal confidential CFO and board-meeting information. Haddix says a similar issue occurred at an unnamed teleconference company.
- The AI security resource hub offers 71 open-source lab links for practicing prompt injection.
Article Details
- Topics
- [00:06] Prompt injection and the claimed real-world basis for the demonstrations
- [01:20] AI penetration testing and red teaming of chatbot access to accounts and data
- [04:14] Prompt-injection taxonomy covering attacker goals, methods, and evasion
- [07:24] Karen CTF lab demonstrates authority impersonation and variable AI responses
- [10:29] Simulated Instagram account-recovery chatbot scenario
- [14:48] Transcript manipulation targets AI-generated meeting summaries
- [17:49] AI security resource hub and open-source practice labs