How Data Poisoning Can Hide Risks in the AI Supply Chain
Summary
An explainer on how poisoned training data and malicious models can introduce hidden AI security risks, why standard benchmarks may miss targeted behavior, and how provenance checks, pipeline controls, red-teaming, and model bills of materials can help.
Key points
- Data poisoning alters training data so a model learns hidden behavior that may persist after deployment; techniques include backdoors, label flipping, and clean-label attacks.
- Attackers may add manipulated content to public sources before those sources are scraped, and research cited in the article found that 250 crafted documents could implant backdoors under specific experimental conditions.
- A backdoored model may perform normally on standard benchmarks while behaving differently when given a targeted trigger.
- AI supply-chain risks can arise in training data, pre-training, model distribution, fine-tuning, repositories, and cloud APIs.
- Malicious executable payloads in model files, such as those found in JFrog's scan of Hugging Face models, are a distinct supply-chain risk from behavioral data poisoning.
- RAG systems face a related but separate risk: manipulated retrieved documents can influence responses through indirect prompt injection without changing the model or its training data.
- Recommended safeguards include verifying model provenance, protecting dataset integrity and access, testing targeted behavior, and documenting model origins and changes in an AIBOM.
Article Details
- Topic
- AI data poisoning and related supply-chain risks, including malicious models and indirect prompt injection