Canary Tokens Can Detect Prompt Extraction but Won’t Prevent It

Summary
The article explains using unique canary strings in LLM context to detect prompt extraction by scanning outputs. It recommends placing markers in different context regions and stresses that canaries detect leaks but do not prevent attacks or catch other exfiltration.
Key points
- Place a unique, high-entropy string in model context and scan responses for it as a signal that the marker was extracted.
- The article suggests separate canaries in system prompts, tool descriptions, and retrieved chunks to help identify which context region leaked.
- Canaries detect the marker in output; they do not stop prompt injection or prevent extraction.
- An output filter could block delivery of a response after detection, but the article notes that the leaked content might otherwise still be delivered.
- The technique will not detect attacks that avoid exposing a marked region, such as tool misuse or other data-exfiltration paths.
- The article cautions that model refusal is not guaranteed and says the output scan is the operative detection step.
Article Details
- Defense Focus
- Detect extraction of protected prompt or retrieved content by scanning model outputs for planted high-entropy canary tokens; this is detection, not prevention.
- Detection Methods
- Generate unique, unguessable canary strings and place them in model context regions.
- Scan each model response for the corresponding canary using an exact substring check.
- Use distinct canaries in different regions to identify which region was exposed.
- Data Sources
- Model responses and output text
- System prompts
- Tool descriptions
- Retrieved RAG chunks
- Worked examples and other prompt regions
- Rule Types
- Exact substring matching against planted canary tokens
- Defensive Actions
- On a canary match, block delivery of the response, alert or page the on-call responder, and rotate the token.
- Use separate canaries for different context regions to help identify the source of extraction.
- Consider concealed canary placements in addition to visible markers.
- Do not rely on canaries to prevent prompt injection or detect attacks that do not expose a marked region; use other controls for those risks.
