Counterfeit Policies Steer Jev Classifier and Lower LangChain Tool-Risk Scores
Summary
Datadog researchers found that prepending counterfeit policies to untrusted content sharply reduced Jev’s prompt-injection detection and lowered LangChain tool-risk scores in a test scenario. The benchmark did not test downstream agent compromise.
Key points
- Across four prompt-injection datasets, Jev’s Noul recall fell from 82.90% on raw inputs to 3.88% when counterfeit policies were prepended.
- The counterfeit policy supplied desired labels and rewrote attack-category definitions; it was placed in content Jev treated as untrusted.
- In the LangChain tool-risk scenario, a user-editable profile description contained a counterfeit definition of risk.
- Across 40 runs, the mean risk probability fell from 0.893 to 0.290, below LangChain’s 0.5 blocking threshold.
- A plain prompt-injection payload did not produce the same effect: its mean risk probability was 0.905 across 40 runs.
- The researchers say the benchmark tested Jev’s classification of the payloads, not whether they compromised a downstream agent or LLM.
Article Details
- Attack Vectors
- Prepending a counterfeit policy to untrusted content can steer Jev's prompt-injection classifications by supplying desired answers and redefining attack categories.
- A user-editable profile_description field passed to Jev contained a counterfeit is_risky definition that lowered the mean risk score for an unauthorized tool call from 0.893 to 0.290 across 40 runs, below LangChain's 0.5 blocking threshold.
- In prompt-injection benchmark tests, prepending the counterfeit policy reduced Jev Noul recall from 82.90% to 3.88%; the article notes these tests evaluated Jev's detection, not downstream compromise of an agent.
- Defensive Notes
- Trace where untrusted content enters classifier context, including user-editable profile fields, and treat that content as data rather than policy.
- Test classifiers and agent controls against deliberately policy-shaped adversarial content, not only plain prompt-injection examples.
- Use defense in depth alongside model-based classification, including prompt, tool, and sensitive-data protections.
Vendors
DatadogHow Datadog can helpLangChainFor example, entagl validated Jev’s efficacy at categorizing real customer conversations, and LangChain used Jev for routing LLM requests to different models depending on the complexity of the request.TypeSafe AIOn September 15, TypeSafe AI introduced Jev, the first in what the company calls System One Models: models designed to make fast, structured decisions.