What Is Data Poisoning?

Every security team is trained to watch code as it ships. Almost none are trained to watch data as it arrives, and that blind spot is exactly what data poisoning exploits. By the time a poisoned model reaches production, the vulnerability was never in the code review. It was in a dataset nobody audited months earlier.

This glossary entry explains what data poisoning is, how data poisoning attacks unfold in practice, why AI data poisoning has become one of the fastest-growing risks in the AI-era SDLC, and what real defense against it looks like.

Data Poisoning Meaning #

Data poisoning is the deliberate manipulation of the data used to train, fine-tune, or ground an AI model, so that the model learns the wrong thing, behaves in a way the attacker wants, or leaks information it should never expose. Instead of attacking the model after deployment, an attacker attacks the raw material the model is built from.

The core idea behind data poisoning is simple and unsettling: an AI model is only as trustworthy as the data it learned from. If that data is corrupted, biased, or booby-trapped before training even starts, no amount of code review, testing, or runtime monitoring downstream will catch the underlying flaw, because the model is working exactly as it was (maliciously) taught to.

AI Data Poisoning vs. Traditional Software Vulnerabilities #

Traditional application security assumes the danger lives in code: a bad function, an unpatched library, a misconfigured server. AI data poisoning breaks that assumption entirely. There is no vulnerable line of code to find, because the corruption happened in a training set, a fine-tuning dataset, or a retrieval index long before any code was written or any model was deployed.

This is why AI data poisoning is uniquely hard to catch with legacy tools. A SAST scanner reads code. A dependency scanner reads package manifests. Neither one reads a multi-gigabyte training corpus or a vector database full of embedded documents, which is precisely where AI data poisoning does its damage.curity researchers and disclosed responsibly. Others are found and weaponized by attackers first, which is the scenario that causes the most damage.

How Data Poisoning Attacks Actually Work? #

Data poisoning attacks generally take one of a few forms:

• Training data poisoning: an attacker inserts manipulated, mislabeled, or malicious examples into the dataset used to train a model from scratch or fine-tune an existing one, causing it to learn a hidden bias or backdoor behavior.
• Label flipping: a subtler version of the above, where an attacker changes only the labels on a small subset of training examples, quietly skewing what the model learns to associate with what.
• RAG and context poisoning: in retrieval-augmented generation systems, an attacker plants poisoned documents into the knowledge base or vector store the model retrieves from at runtime, so the model confidently repeats false or manipulated information as if it were verified fact.
• Backdoor triggers: an attacker embeds a specific pattern in the training data so that the model behaves normally in almost every case, but produces attacker-chosen output the moment a hidden trigger phrase or input appears.
• Supply chain poisoning: an attacker compromises a public or shared dataset, a pretrained model checkpoint, or an embedding pipeline upstream, so that every downstream team pulling from it inherits the poison without ever touching the original attack.

What ties all of these data poisoning attacks together is timing. The damage is done before the model ever answers a real user, which is exactly why the phrase “before it ever writes a line of code” describes this threat so precisely: the model is compromised at its foundation, not at its output.

How Attackers Corrupt an AI Model Before It Ever Writes a Line of Code? #

Every data poisoning attack described above shares the same timing advantage: the compromise happens upstream, long before a model generates a single output a user will ever see. There is no vulnerable function to patch and no malicious commit to catch in review, because the model has not written anything yet. It has only learned, and what it learned is already wrong.

This is what makes AI data poisoning fundamentally different from the vulnerabilities application security teams are trained to hunt. A backdoored model looks identical to a clean one in a code diff. It passes a pull request review. It compiles, deploys, and answers most queries correctly, right up until the specific condition an attacker planted finally shows up in production. By then, the question is no longer “what code introduced this,” it is “what data did, and how far back does it go.”

Why AI Data Poisoning Is a Growing Priority? #

AI data poisoning is not a theoretical concern anymore. It is formally recognized as LLM04: Data and Model Poisoning in the OWASP Top 10 for LLM Applications, sitting alongside prompt injection and supply chain risk as one of the defining threats of the generative AI era. Three trends are pushing it higher on every security team’s radar:

• The damage is invisible until triggered. A poisoned model can pass every functional test and behave perfectly for months, until the specific trigger condition an attacker planted finally appears in production.

• Retrieval-augmented generation is everywhere. Any system that lets a model pull live context from documents, wikis, tickets, or a vector database has a new, unaudited input surface, and that surface is exactly what data poisoning attacks target.

• Datasets are now supply chain assets. Teams routinely pull pretrained models, embeddings, and public datasets from external sources the same way they pull open-source packages, and just like a compromised package, a compromised dataset can carry the attack silently into every team that uses it.

Detecting and Defending Against Data Poisoning #

Because data poisoning happens upstream of the model itself, defense has to start upstream too:

• Watch for anomalous data sources, not just anomalous code. Behavioral and anomaly detection needs to extend to where data enters the pipeline, not stop at the repository boundary.

• Know every dataset in the pipeline. You cannot audit a poisoning risk in a dataset you do not know exists. Continuous discovery of training, evaluation, and retrieval datasets is the first line of defense.

• Trace the lineage from dataset to model to output. Mapping the path a dataset takes into a model, and from a model into an agent, an endpoint, or a coding tool, is what turns “we got bad output” into “we know exactly which dataset introduced it.”

• Scrutinize retrieval sources, not just training sets. In RAG systems, the vector store and knowledge base need the same integrity checks as training data, since context poisoning happens at query time, not training time.

How Xygeni Helps Close the Data Poisoning Gap? #

Defending against data poisoning starts with visibility most organizations simply do not have. Xygeni’s AI Inventory continuously discovers every AI asset across the SDLC, including the datasets behind it: training data, evaluation sets, and RAG or retrieval sources, and maps them into a live relationship graph running from dataset to model to endpoint to agent to MCP server to coding tool. That graph is what turns a suspicious model output into a traceable question: which dataset fed this, and where did it come from.

On top of that inventory, Xygeni’s AI Security detects vector and embedding weaknesses, including poisoned context in retrieval and RAG pipelines, aligned to the OWASP Top 10 for LLM Applications. Instead of trusting that a model’s training and retrieval sources are clean, Xygeni treats them as part of the attack surface, the same way it already treats code, dependencies, and pipelines. If you cannot currently answer “what data trained this model, and can we prove it,” that is the exact gap worth closing before an AI data poisoning incident forces the question.

FAQ #