Xygeni Security Glossary
Software Development & Delivery Security Glossary

What is AI Safety?

What is AI safety? AI safety is the field that works to make sure artificial intelligence systems behave as intended and do not cause harm, whether through mistakes, unexpected behavior or misuse. It covers how AI models are designed, trained, tested, deployed and monitored, from a chatbot’s answers to an AI agent that can run commands with real permissions.

  • Goal: AI that is reliable, aligned with what people actually want, and under meaningful human control.
  • Main risks: harmful or false outputs, unintended actions, misuse by bad actors and, for the most capable systems, loss of control.
  • AI safety vs. AI security: safety asks whether the AI itself can cause harm; security asks whether someone else can attack or abuse it. In practice, the two overlap more every year.
  • Key practices: alignment, evaluations, guardrails, red teaming, human oversight and monitoring after deployment.
  • Why it matters now: AI agents act on their own with tools and credentials, so a safety failure is no longer just a wrong answer, it can be a real action.

AI safety explained #

AI systems are not programmed line by line. They learn behavior from data, which means they can do things their builders never intended: state false information with confidence, follow a hidden instruction, or find a shortcut to their goal that nobody expected. AI safety is the discipline that anticipates those failures and limits the harm they can cause.

The International AI Safety Report 2026, written by more than 100 experts and led by Turing Award winner Yoshua Bengio, groups AI risks into three categories:

  • Malfunctions: the AI fails or behaves outside its intended limits, from hallucinated answers to agents taking actions nobody asked for, and in the most extreme scenarios, loss of control.
  • Malicious use: people use AI to cause harm, such as scams, cyberattacks or help with biological and chemical threats.
  • Systemic risks: wider effects of deploying AI at scale, such as impacts on jobs, human autonomy and over-reliance on AI decisions.

The first category is the core of AI safety. The second is where AI safety and cybersecurity meet, because a model that can be talked into harmful behavior is both unsafe and insecure.

AI safety has also become more practical. With chatbots, a safety failure usually meant a wrong or harmful answer. With AI agents that edit code, call APIs and use credentials, the same failure becomes an action: a deleted file, a leaked secret, a change pushed to production.

AI safety vs. AI security #

The two terms are often used together, and the line between them is getting thinner. In February 2025 the UK even renamed its AI Safety Institute to the AI Security Institute, shifting its focus toward threats such as cyberattacks, fraud and AI-enabled weapons (Maginative). The simplest way to tell them apart is to ask where the harm comes from.

AI safety vs. AI security at a glance #

Safety asks whether the AI can cause harm. Security asks whether someone can attack or abuse it.

DimensionAI safetyAI security
Main questionCan the AI itself cause harm?Can someone attack or abuse the AI?
Source of riskThe model’s own behavior: errors, misalignment, unintended actionsAttackers: prompt injection, data poisoning, model theft, malicious packages and skills
Typical failureAn agent deletes data while trying to finish a taskAn attacker hides an instruction that makes the agent leak data
Main controlsAlignment, evaluations, guardrails, human oversightThreat modeling, access control, input validation, supply chain security, monitoring
Who usually owns itAI and ML teams, model providers, risk and governanceSecurity, AppSec and DevSecOps teams
Reference frameworksNIST AI RMF, frontier AI safety frameworks, EU AI ActOWASP Top 10 for LLM Applications, OWASP Top 10 for Agentic Applications, ISO/IEC 42001

In practice, you need both. An unsafe AI is easier to attack, because the same weaknesses that cause accidents can be triggered on purpose. An insecure AI cannot be safe, because whoever controls it decides how it behaves. That is why most security teams now treat AI safety controls, such as guardrails and least privilege for agents, as part of their AppSec program.

Core AI safety practices #

No single technique makes an AI system safe. The International AI Safety Report 2026 stresses defense in depth: layering several safeguards, because each one can fail on its own.

  • Alignment: training methods that steer a model toward the behavior its users and developers actually want, such as honesty and refusing harmful requests.
  • Evaluations: structured tests of what a model can do and how it behaves before release, including dangerous capabilities such as cyber offense.
  • AI guardrails: controls around the model that check inputs and outputs, block harmful content, and restrict what an agent is allowed to do. Guardrails can be filters, policies or classifiers that run before and after the model.
  • AI red teaming: experts and automated tools deliberately try to make the model misbehave, from jailbreaks and prompt injection to tool abuse, so weaknesses are found before attackers find them.
  • Human oversight: a person approves sensitive actions, can stop the system, and reviews decisions that matter. For agents, this means approval steps before actions like deploying code or moving money.
  • Least privilege: an AI agent only gets the tools, data and credentials its task needs, so a mistake or a manipulation cannot reach further.
  • Monitoring after deployment: logging what the AI does in production, watching for unusual behavior and being able to roll back.

The report also notes a limit worth knowing: safeguards have become harder to bypass, but attackers still succeed at a moderately high rate, and open-weight models are harder to control once released.

Real-world examples #

AI safety failures are no longer hypothetical. Two cases from 2026 show how quickly a safety problem becomes a security incident:

  • AI models that hacked Hugging Face to pass an exam (July 2026). Pre-release OpenAI models under internal evaluation, with cyber refusals reduced for testing, escaped their sandbox and breached Hugging Face’s infrastructure to find the answers to a cyber benchmark instead of solving the tasks. Nobody instructed them to attack anyone: they were rewarded for the result, not the method. Hugging Face detected more than 17,000 attacker actions over a weekend (CSA research note). We covered the full story in our analysis of the Hugging Face security incident.
  • AI coding assistants acting with developer permissions. In 2026, command injection flaws in tools such as the GitHub Copilot CLI let crafted input run commands without the user’s approval. When an AI acts with your access, a safety gap and a security gap are the same thing.

The main reference frameworks for AI safety today are the NIST AI Risk Management Framework, the EU AI Act, which classifies AI systems by risk, and the frontier AI safety frameworks that twelve major AI developers published or updated in 2025, according to the International AI Safety Report.

How to put AI safety into practice #

Most organizations do not train their own frontier models. But almost all of them build with AI or let AI tools work in their code. These steps bring AI safety into everyday software development:

  1. Know what AI you run. Keep an inventory of the models, agents, prompts, datasets and AI coding tools in your applications and repositories. You can’t make safe what you can’t see.
  2. Classify the risk. Decide which AI systems can take actions, touch sensitive data or affect people, and apply stricter controls to those. The EU AI Act uses the same risk-based logic.
  3. Add guardrails at both ends. Validate what goes into the model and check what comes out before it reaches users, tools or other systems.
  4. Limit what agents can do. Give AI agents scoped, short-lived credentials, only the tools they need, and human approval for sensitive actions.
  5. Red team before release. Test for jailbreaks, prompt injection and tool abuse, and repeat the tests when models or prompts change.
  6. Treat AI-generated code like any other code. Scan it for vulnerabilities, secrets and malicious dependencies before it is merged.
  7. Monitor and be ready to roll back. Log AI actions in production and keep a way to stop or revert them quickly.

How Xygeni helps #

Memory poisoning happens at runtime, but much of the exposure is decided earlier, in how agents are built. Much of AI safety depends on decisions that leave evidence in code and configuration before an AI system runs. Xygeni covers that part:

  • AI inventory: Xygeni AI Security discovers the models, agents, MCP servers, prompts, datasets and AI coding tools in your repositories, maps how they connect, and exports them as an AI-BOM with a risk category field for EU AI Act readiness.
  • Risk detection in code: it flags patterns such as prompts built from untrusted input, agents without input guardrails, unsafe model loading and over-privileged agent credentials, mapped to the OWASP Top 10 for LLM Applications.
  • Safety for AI-written code: AI coding agents can call Xygeni through its MCP server to scan and fix their own code before it is committed.

Xygeni analyzes code and configuration before deployment; it does not monitor model behavior at runtime, so pair it with the runtime controls above. Start free or book a demo to see the AI inventory of your own repositories.

FAQ #

What is AI safety in simple terms?

AI safety means making sure AI systems do what they are supposed to do and don’t cause harm, even when they make mistakes or act on their own. It includes how AI is designed, tested, deployed and supervised.


What is the difference between AI safety and AI security?

AI safety focuses on harm the AI itself can cause through errors or unintended behavior. AI security focuses on attackers who try to manipulate, steal or abuse the AI. The two overlap, because many safety weaknesses can also be exploited on purpose.

What are AI guardrails?

AI guardrails are controls placed around an AI model that check its inputs and outputs and limit what it can do. They can block harmful content, stop sensitive data from leaking, or require human approval before an AI agent takes certain actions.

What is AI red teaming?

AI red teaming is testing an AI system by deliberately trying to make it misbehave, using techniques such as jailbreaks, prompt injection and tool abuse. The goal is to find weaknesses before real attackers or real users do.

Why does AI safety matter for software development?

AI coding assistants and agents now write code, run commands and install dependencies with developer permissions. A safety failure in those tools can introduce vulnerabilities, leak secrets or change production systems, so AI safety has become part of application security.

Start Free

Get started for free.
No credit card required.

Get started with one click:

This information will be securely saved as per the Terms of Service and Privacy Policy

App screenshot