Python Demos Show How Adversarial Inputs Can Manipulate AI Models and Agents

Summary

AI security expert Harriet demonstrates image perturbations and a simulated prompt-injection cascade, and explains risks including agent memory poisoning and timing side channels.

Key points

• A Python demonstration altered an image so a model classified a visible rifle as a bassoon with 99% confidence; a targeted variation produced a starfish classification.
• Harriet distinguished manipulation of inputs to a pretrained model from poisoning its training data.
• A simulated support-ticket system showed how a prompt injection could propagate between agents when validation and monitoring are absent.
• She described memory poisoning as storing a malicious instruction for possible execution after a later trigger.
• She explained that model prediction timing could potentially reveal a classification if an attacker has sufficiently sensitive measurements and access to relevant information.
• The guest noted that adversarial patches can be difficult to apply reliably in real-world conditions and are ideally crafted for a specific target model.

People