Data poisoning

Data poisoning attacks the learning or retrieval process rather than the model’s inference at runtime. In a training-time attack, an adversary introduces malicious examples into the dataset used to train or fine-tune a model. The model learns an association the attacker intended: a specific trigger phrase causes a particular output, a class of inputs gets systematically misclassified, or the model develops a preference for outputs that benefit the attacker.

Retrieval-augmented generation (RAG) systems, which ground model outputs in a knowledge base, introduce a retrieval-time variant. If an attacker can write to the knowledge base (a shared wiki, a document repository, a website the system indexes), they can plant content designed to be retrieved in response to certain queries and influence the model’s outputs without touching the model weights at all. The model faithfully summarizes the poisoned source.

Defense is difficult because the attack surface varies with the data pipeline. For training-time poisoning, controls focus on provenance and integrity: where training data comes from, who can contribute to it, and whether anomalous examples are flagged before ingestion. For retrieval-time poisoning, the relevant controls are access management on the knowledge base, source attribution in model outputs so users can verify claims, and monitoring for outputs that diverge from expected behavior. The governance question of who can write to an AI system’s data sources is as important as the technical controls.