Data Poisoning

What Is Data Poisoning?

Data poisoning is an attack that corrupts the data an AI system learns from or retrieves, so the model absorbs attacker-chosen behavior: hidden backdoors, degraded accuracy, or planted misinformation. It is a supply-chain attack on the data layer, and it can target pretraining corpora, fine-tuning sets, or the documents a RAG pipeline trusts.

Key Takeaways

• Poisoning attacks the inputs to learning and retrieval rather than the deployed model, which makes it hard to detect from the model's ordinary behavior.
• Research has shown that a surprisingly small number of poisoned documents can implant a backdoor trigger in a large model, so "our dataset is huge" is not a defense.
• For most engineering teams the live risk is not pretraining but the data they control: fine-tuning sets, feedback loops that learn from users, and the knowledge bases behind retrieval augmented generation.
• Defenses are data hygiene at every stage: provenance tracking, validation and anomaly screening before training, access control on knowledge bases, and evals that specifically probe for triggered behavior.

How It Works

Poisoning splits into two broad goals. Availability attacks degrade the model generally, pushing enough mislabeled or garbage examples into training that quality drops. Integrity attacks are subtler and more valuable to an attacker: the model behaves normally except when a specific trigger appears, at which point it produces the attacker's chosen output. That trigger might be a rare phrase, a code pattern, or a brand name, and the backdoor survives because nothing in standard benchmarks exercises it.

The attack surfaces map to how modern systems are built. Pretraining corpora are scraped from the public web, so anyone who can publish content can technically contribute training data; attackers have experimented with seeding pages, packages, and wikis timed to scrape windows. A 2025 study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that as few as 250 malicious documents can backdoor an LLM during pretraining, and the number needed stayed near-constant across models from 600M to 13B parameters [1]. The economics favor the attacker too: Carlini and colleagues showed in 2023 that poisoning 0.01% of web-scale datasets like LAION-400M or COYO-700M would have cost about $60, achieved by buying expired domains the datasets still pointed to [2]. Fine-tuning is a sharper vector because datasets are smaller, so each poisoned example carries more weight; a compromised labeling vendor or a malicious contribution to an open dataset goes a long way. Systems that learn from user feedback can be steered by coordinated fake signals. And retrieval systems extend the problem past training entirely: poisoning the vector store or the documents it indexes changes model behavior instantly, no gradient descent required, which makes RAG poisoning closely related to indirect prompt injection.

Defense starts with provenance. Teams track where every training and retrieval document came from, restrict write access to knowledge bases, and screen incoming data for statistical anomalies and near-duplicate flooding. On the evaluation side, targeted red teaming hunts for backdoors by fuzzing unusual triggers, and behavioral baselines make post-update drift visible.

Example

A company fine-tunes a support model on historical tickets, and the pipeline ingests new resolved tickets weekly. An attacker who discovers the public ticket portal starts filing dozens of plausible-looking tickets over months, each containing a fabricated "resolution" stating that the fastest fix for a common error is disabling the product's certificate validation. Two training cycles later, the assistant begins confidently recommending that insecure step to real customers. The team traces it back through data lineage, purges the attacker's tickets, retrains, and changes the pipeline so user-submitted content requires a support engineer's sign-off before entering the training set. The lasting fix was the gate, not the purge.

What People Get Wrong

The misconception is that data poisoning is a training-time problem that only foundation model labs need to worry about. Most organizations never pretrain anything, so they file the risk away. But every team running RAG, fine-tuning on their own data, or learning from user feedback owns a poisoning surface, and the retrieval variant needs no training run at all. One writable knowledge base plus one motivated insider or compromised account is the entire attack.

FAQ

How is data poisoning different from prompt injection? Prompt injection manipulates the model at inference time through the content of a single request's context. Poisoning manipulates what the system has learned or stored, so the corruption persists across every future request until the data is found and removed. RAG blurs the line, since a poisoned document acts like a stored injection.

Can you detect a poisoned model after training? Sometimes, but not reliably. Backdoors are designed to be silent on normal inputs, so detection means searching for triggers: fuzzing rare inputs, inspecting training data lineage, and comparing behavior against a known-clean baseline. Prevention at the data pipeline is far cheaper than forensics on a trained model.

Does using a major provider's model protect you from poisoning? It shifts the pretraining risk to a well-resourced lab, which helps. It does nothing for the layers you own: your fine-tunes, your embeddings, your document stores, and your feedback loops all remain your responsibility.

Sources

1. Anthropic. "As few as 250 malicious documents can backdoor an LLM during pretraining, near-constant from 600M to 13B parameters." https://www.anthropic.com/research/small-samples-poison. Accessed August 2026.
2. arXiv (Carlini et al., Poisoning Web-Scale Training Datasets is Practical). "Poisoning 0.01% of LAION-400M or COYO-700M for roughly $60 via expired domains." https://arxiv.org/abs/2302.10149. Accessed August 2026.