TL; DR
What a model extraction attack is, and why inventory comes first
A model extraction attack copies your AI through its own API. No breach, no stolen weights, no malware.
- איך זה עובד: a model extraction attack (also called model stealing or model theft) queries an inference endpoint at scale and trains a clone on the answers. To your logs, the attacker looks like a heavy customer.
- Where it sits: OWASP classifies model extraction under LLM10:2025 Unbounded Consumption and lists a centralized model inventory among its mitigations.
- Why defenses miss: rate limits, authentication and monitoring only protect the endpoints you know about. Shadow AI endpoints get none of them.
The first defense: an AI inventory of every model, agent and inference endpoint your code exposes, discovered from source before it ships. You can't protect inference endpoints you haven't inventoried.
Your model can be stolen without anyone touching your servers
Your team spent months fine-tuning a model on proprietary data. It sits behind an API that answers thousands of requests a day. Now imagine a competitor rebuilding most of its value for the price of those API calls.
That is a model extraction attack. Nobody breaches your infrastructure or downloads your weights. The attacker simply asks your model enough well-chosen questions, records the answers and trains a copy.
Most teams answer this risk with rate limits, authentication and output monitoring. Those controls work. But they share one blind spot: they only protect the inference endpoints you know exist. In this guide we explain what a model extraction attack is, how model stealing works in practice, and why every defense starts with an inventory.
What is a model extraction attack?
A model extraction attack is an attack in which an adversary reconstructs a functional copy of a machine learning model by sending it queries and learning from its outputs. It is also known as model stealing or model theft. The attacker never needs the original weights, training data or source code: the public interface is enough.
The idea is not new. Researchers showed it against commercial prediction APIs in 2016 (Tramèr et al., USENIX Security). In 2024, a team at Google DeepMind recovered part of a production language model through its public API (Carlini et al.). אטלס מיטרה tracks the same behavior as an adversary technique against AI systems.
For LLM applications, OWASP folds model theft into LLM10:2025 Unbounded Consumption. It describes two variants: extraction via crafted API queries, and functional replication, where the target model generates synthetic data to fine-tune a competing one.
Model extraction attack: key terms
The same threat goes by several names. Here is what each one means, and where they overlap.
| טווח | מה זה אומר |
|---|---|
| Model extraction attack | Rebuilding a model's behavior from its query responses |
| Model stealing / model theft | Common synonyms; model theft also covers direct theft of weight files |
| Functional model replication | Using the target's outputs as training data for a substitute model |
| Shadow model | The attacker's clone, trained to imitate the target |
How a model extraction attack works
Every model extraction attack follows the same loop: find an endpoint, query it at scale, train a copy.
- Find the endpoint. The attacker locates an inference API: a chat backend, a classification service, an embeddings endpoint, an internal model a partner can reach.
- Design the queries. Inputs are chosen to cover the model’s decision space efficiently, sometimes combined with prompt injection to coax out richer answers.
- Harvest the outputs. Labels, confidence scores, logits, or full text completions all leak information. The richer the response, the faster the theft.
- Train the substitute. The collected input and output pairs become a training set for a shadow model that imitates the original.
- Exploit the clone. The copy is used to skip licensing costs, compete directly, or rehearse adversarial attacks offline before firing them at the real model.
What makes model stealing hard to catch is that every single request is legitimate. The signal is in the volume, the pattern and, above all, in where the requests land.
Why the usual defenses miss: the inventory gap
השמיים standard advice against a model extraction attack is sound: authenticate callers, rate-limit and throttle, trim confidence scores from responses, watermark outputs, and watch for anomalous query patterns. Each control is applied per endpoint. That is exactly where it breaks.
AI now enters codebases faster than security teams can track it. A developer wires an LLM SDK into a microservice. A data team exposes a fine-tuned model for an internal tool. An agent framework spins up its own serving layer. None of these go through a review queue, and many never appear in an API gateway or a spec file.
The result is shadow AI: models, agents, and inference endpoints that exist in production but not on anyone’s list. An endpoint nobody knows about gets no rate limit, no authentication review, and no monitoring. For a model extraction attack, it is the ideal target.
OWASP points the same way. Among its mitigations for LLM10, it recommends maintaining a centralized inventory of ML models. The logic is simple: you can’t protect inference endpoints you haven’t inventoried.
Inventory first: discover every endpoint, then protect it
A useful AI inventory answers four questions for each asset: what it is, where it is declared, what it talks to, and what protects it.
| שאלת מלאי | Why it matters against model extraction |
|---|---|
| Which models and inference endpoints exist? | Every unlisted endpoint is an unprotected extraction target |
| Where are they declared in code? | Ownership: someone must fix the missing control |
| What data and tools are connected? | A model trained on sensitive data is worth more to steal |
| איזה guardrails and controls apply? | Shows coverage gaps before an attacker finds them |
The best place to build this inventory is in source code. Code is where an endpoint is born, so discovering AI assets in the repository finds them before they reach production. It also links each asset to a project, a commit, and a team, which is what turns a list into action.
Two more habits close the loop. Export the inventory as an AI bill of materials (AI-BOM) so governance and audit teams see the same picture. And rescan on every change, because the inventory is only as good as its last update.
How Xygeni closes the gap before a model extraction attack starts
אבטחת בינה מלאכותית של Xygeni builds the AI inventory directly from source code. It discovers AI models, agents, AI frameworks, MCP servers and tools, prompts, guardrails, vector stores, datasets, inference endpoints and AI coding tools across your repositories, including model files such as .onnx, .pt ו .pkl. Relationships are mapped too: which agent uses which model, which tools it executes, which datasets it trains on.
Each asset in the AI Assets inventory shows its models and providers, connected data sources with PII flags, guardrail coverage, and governance fields such as the EU AI Act risk category. The AI Graph view shows how agents, models, tools, and datasets connect. One-click exports a CycloneDX 1.6 ML-BOM, next to your SBOM and CBOM.
From inventory to protection:
- Risk detection mapped to OWASP. Detector packs cover all ten OWASP LLM Top 10 categories, including LLM10 Unbounded Consumption, so every finding speaks the language your auditors use.
- Endpoint controls checked in code. אבטחת API של Xygeni discovers API endpoints from code and specs and flags unauthenticated endpoints, missing rate limiting, and shadow APIs that drifted from the spec.
- Proof on the running app. Xygeni AI Pentesting tests the deployed application and proves which weaknesses are exploitable.
- Fix fast. AI Triage prioritizes AI findings and AI Remediation generates fix guidance for all ten OWASP LLM categories.
All of it runs as a static scan in CI/CD or locally, with SaaS, on-premise and air-gapped options. Teams that need the semantic review stage to stay in-house can run it on a self-hosted model.
Model extraction attack prevention checklist
Eight controls, in order. The first two make the other six possible.
- מלאיInventory every model, agent and inference endpoint from source code, and keep it current on every commit.
- בעלותAssign an owner to each AI asset in the inventory.
- אימותRequire authentication on every inference endpoint, internal ones included.
- גבול דירוגApply rate limits, quotas and timeouts per caller.
- Output minimizationReturn the minimum output needed: drop confidence scores and logits where users don't need them.
- ניטורMonitor query volume and patterns for systematic probing.
- watermarkingConsider watermarking outputs of high-value proprietary models.
- ממשלExport an AI-BOM for governance and audit, and map findings to OWASP LLM10.
Find your inference endpoints before attackers do
A model extraction attack only needs one endpoint you forgot about. Find out what AI your code really runs: see how Xygeni AI Security builds your AI inventory, or הזמן הדגמה.
שאלות נפוצות
What is the difference between a model extraction attack and model inversion?
A model extraction attack copies the model itself: its behavior and decision logic. Model inversion targets the training data, trying to reconstruct sensitive records the model learned from. Both abuse query access, but extraction steals intellectual property while inversion leaks private data.
Can a model extraction attack target LLMs, or only classic ML models?
Both. Early research targeted classifiers behind prediction APIs. Today OWASP lists model extraction and functional model replication as LLM risks under LLM10:2025 Unbounded Consumption, where an attacker uses an LLM’s own outputs to fine-tune a substitute.
How do I know which of my inference endpoints are exposed to model stealing?
Start with an inventory built from source code: every model, agent, and inference endpoint your applications declare. Then check each one for authentication, rate limiting, and output exposure. Xygeni AI Security discovers AI assets across your repositories and exports them as an AI-BOM, and Xygeni API Security flags endpoints that lack authentication or rate limiting.
