The AI Security Glossary

Active learning is a training approach where the model flags the examples it is least sure about so humans label those first, cutting how much labelled data is needed.

An AI agent is an AI system that plans and executes multi-step tasks on a user's behalf, calling tools such as browsers, code interpreters, and APIs rather than only generating text.

Artificial general intelligence (AGI) is a hypothetical AI system able to match or exceed human performance across most cognitive tasks, rather than excelling only in a narrow domain.

Artificial narrow intelligence (ANI) is AI built to do one kind of task well, such as spam filtering or image tagging, rather than reasoning across domains like a human.

Average precision is a single score that summarises how well a model ranks correct results ahead of wrong ones across all thresholds, used to compare detection and classification models.

Backpropagation is the core algorithm that trains a neural network by working backwards from its errors to adjust each internal weight so the next prediction is a little more accurate.

Bayes' theorem is a rule for updating the probability that something is true as new evidence arrives, combining prior belief with fresh data.

A Bayesian network is a model that maps how uncertain factors influence each other, so you can reason about the probability of one event given what you know about the others.

A context window is the maximum amount of text, measured in tokens, that a language model can consider at once when producing a response.

A neural network built for grid-shaped data such as images, which learns its own visual features by sliding small filters across the input.

A model that reaches a decision by asking a sequence of yes/no questions about the data, splitting it at each step until it arrives at an answer.

An embedding is a numeric vector that represents the meaning of text, images, or other data, so that similar items sit close together in mathematical space.

Explainability is the ability to answer how an AI system arrived at a particular result, in terms the person receiving the answer can actually act on.

Feature attribution assigns each input a share of the responsibility for a particular model output. It is a reconstruction made after the fact, not a record of what the model did.

Fine-tuning is the process of further training an existing AI model on a smaller, specialized dataset so it performs better on a specific task or domain.

A frontier model is one of the most capable general-purpose AI models available at a given time, typically built by a small number of well-funded labs.

Generative AI is a class of artificial intelligence that creates new content, including text, images, audio, video, and code, rather than only classifying or ranking existing data.

A hallucination is an AI output that is fluent and confident but factually wrong or entirely invented.

Inference is the phase in which a trained AI model is actually used: it receives an input and produces an output, as opposed to the training phase in which the model learns.

An unsupervised method that sorts data into a number of groups you choose in advance, assigning each record to whichever group centre it sits nearest.

A large language model (LLM) is an AI model trained on massive amounts of text to predict the next token, which enables it to generate and work with language fluently.

A model that predicts a number by fitting a straight-line relationship to the data, weighting each input by a coefficient you can read and check.

Building software that learns patterns from data rather than following only rules a person wrote, so its behaviour comes from the examples it was trained on.

The Model Context Protocol (MCP) is an open standard, introduced by Anthropic in 2024, that lets AI applications connect to external tools and data sources through a common interface.

Multicollinearity is two or more model inputs carrying substantially the same information, leaving the model no basis for deciding which one earned the credit for a prediction.

Multimodal AI is an AI system that can process or generate more than one type of data, such as text, images, audio, and video, within a single model.

A model built from layers of simple connected units whose weights are adjusted during training, letting it learn relationships too complex to write down.

An open-weights model is an AI model whose trained parameters are published for anyone to download, run, and fine-tune, unlike closed models accessible only through a vendor's API.

Prompt engineering is the practice of writing and structuring inputs to an AI model to get reliably useful outputs.

Reinforcement learning from human feedback (RLHF) is a training technique that tunes an AI model toward outputs humans rate as helpful and safe.

Retrieval-augmented generation fetches relevant documents from a knowledge base and inserts them into the model's prompt, so answers are grounded in current, private, or authoritative data.

Training a model on examples that have already been labelled with the right answer, so it learns to reproduce those labels on data it has not seen.

Synthetic data is artificially generated data that mimics the statistical properties of real data, used for training or testing when real data is scarce, sensitive, or regulated.

A system prompt is the hidden instruction block an operator gives an AI model to define its role, rules, and constraints before any user input arrives.

A token is the basic unit of text an AI language model reads and writes, typically a word fragment of about four characters in English.

Training data is the corpus of examples an AI model learns from, which determines what the model knows, how it behaves, and what biases and secrets it may carry.

The transformer is the neural network architecture, introduced by Google researchers in the 2017 paper 'Attention Is All You Need', that underlies virtually all modern large language models.

Finding structure in data that carries no labels, so the model groups or ranks what it sees without being told in advance what any of it means.

An AI gateway is a proxy that sits between an organization's users or applications and external AI models, enforcing policy on every prompt and response.

AI observability is the practice of monitoring an AI system in production: its inputs, outputs, performance, and drift, so you can tell when it starts behaving in ways you did not intend.

The AI supply chain is everything an AI system depends on that you did not build: base models, training data, fine-tuning datasets, open-source libraries, model hubs, plugins, and hosted APIs.

Data leakage, in an AI context, is the unintended exposure of sensitive information through an AI system: data pasted into prompts, revealed in model outputs, or absorbed into training sets.

Corrupting the data an AI model trains or retrieves from so that the model learns, retrieves, or repeats attacker-chosen behavior.

A deepfake is synthetic audio, video, or imagery generated by AI to convincingly impersonate a real person.

Excessive agency is the failure mode where an AI system is granted more permissions, tools, or autonomy than its task requires, so a mistake or a hijacked prompt causes real-world damage.

Guardrails are the technical controls wrapped around an AI model to constrain what it will accept and produce, separate from the model's own training.

A jailbreak is a prompt or technique that manipulates an AI model into ignoring its safety rules and producing output it was trained to refuse.

Model extraction is an attack that reconstructs a proprietary AI model, or the sensitive data behind it, by systematically querying it and studying the responses.

Model poisoning is tampering with an AI model itself, during training, fine-tuning, or distribution, so that it carries hidden malicious behavior.

The OWASP Top 10 for LLM Applications is a community-maintained list of the most critical security risks in applications built on large language models.

An attack that smuggles instructions into an AI system's input so the model follows the attacker's intent instead of the operator's.

AI red teaming is the structured adversarial testing of an AI system to find harmful, insecure, or policy-violating behavior before attackers or users do.

Shadow AI is the use of AI tools by employees or teams without the knowledge or approval of IT and security.

An AI acceptable use policy is an internal policy that defines which AI tools employees may use, for what tasks, and with what categories of data.

An AI ethics board is a cross-functional group that reviews proposed AI uses against the organization's values and risk appetite, especially where they affect people.

An AI fallback plan is the documented way of operating when an AI system must be turned off or bypassed: the manual or alternative process that keeps things running.

AI governance is the set of policies, roles, and review processes an organization uses to decide how AI systems are acquired, built, deployed, and monitored.

An AI governance charter is the founding document that sets who decides what about AI in an organization: the roles, the scope, the decision rights, and the escalation paths.

An AI impact assessment is a structured review of an AI use case's risks, benefits, and affected parties, run before deployment to inform the decision to proceed.

AI Trust, Risk and Security Management: an analyst umbrella term for the tooling and practices that keep deployed AI explainable, monitored, secured and compliant.

The EU AI Act is the European Union's regulation for artificial intelligence, in force since August 2024, which imposes obligations on AI systems in tiers based on risk.

A fundamental rights impact assessment (FRIA) is an EU AI Act requirement for certain deployers of high-risk AI to assess the system's impact on people's rights before use.

Human-in-the-loop is a design pattern in which a person reviews, approves, or can override an AI system's output before it takes effect.

ISO/IEC 38507 is an international standard on the governance implications of using AI, written for the board and governing body rather than the technical team.

ISO/IEC 42001 is the international standard, published in 2023, for AI management systems: a certifiable framework for governing how an organization develops and uses AI responsibly.

A model owner is the named person accountable for a specific AI model in production: its performance, its risks, and the decision to retrain, roll back, or retire it.

A model risk committee is the governance body that reviews and approves AI models against risk criteria before deployment and on a recurring basis afterwards.

Model rollback is reverting a production AI model to a previous version known to work, usually because the current one has failed quality, drift, or safety checks.

The NIST AI Risk Management Framework (AI RMF) is a voluntary United States framework, published in January 2023, that helps organizations identify, measure, and manage risks from AI systems.

Post-market monitoring is the ongoing tracking of an AI system's performance and risks after deployment, an explicit obligation for high-risk systems under the EU AI Act.

Provider vs deployer is the EU AI Act's split between the party that builds or substantially modifies an AI system and the party that puts it to use, each carrying different duties.

A serious AI incident is a malfunction or harm from an AI system that meets a defined severity bar, which the EU AI Act requires providers to report within strict deadlines.

ChatGPT is OpenAI's conversational AI assistant, launched in November 2022, and the product that took large language models mainstream.

Claude is the AI assistant family built by Anthropic, an AI safety focused company founded in 2021 by former OpenAI researchers.

Cohere is an enterprise AI company, founded in 2019 in Toronto, that builds language models aimed at business use rather than consumer chat.

Copilot is Microsoft's brand for the AI assistants embedded across Windows, Microsoft 365, and GitHub, largely powered by OpenAI models.

Gemini is Google's family of multimodal AI models and assistants, developed by Google DeepMind and integrated across Google Search, Workspace, and Android.

Hugging Face is the largest open platform for sharing machine learning models and datasets, founded in 2016 and often described as the GitHub of AI.

Midjourney is an AI image generation service, run by a small self-funded research lab, known for producing high-quality stylized images from text prompts.

Mistral AI is a French AI company, founded in 2023, that builds high-performing language models and releases many of them with open weights.

Perplexity is an AI answer engine, founded in 2022, that responds to questions with synthesized answers and cited sources rather than a list of links.

Fraud that uses a trusted email identity, forged or genuinely compromised, to make a legitimate employee authorize a payment or hand over data.

DomainKeys Identified Mail: a domain signs each message and publishes the verifying key in DNS, so a receiver can confirm the message is intact and that a named domain vouched for it.

A DNS policy record at _dmarc that tests whether an SPF or DKIM pass belongs to the domain in the From: header, publishes an enforcement instruction, and requests reports.

Endpoint Detection and Response: an agent that records endpoint activity, detects suspicious behavior, and lets responders isolate or remediate machines remotely.

Forging the sender identity on an email so it appears to come from a trusted domain or person, which base email does nothing to prevent.

The principle that every user, process, and system gets exactly the access it needs to do its job, and nothing more.

The property that someone cannot credibly deny an action they took, because evidence binds that specific person to that specific act.

Fraudulent messages that impersonate a trusted party to trick people into revealing credentials, paying money, or running malware.

Cryptographic algorithms designed to resist attack by quantum computers, replacing RSA and ECC before large-scale quantum machines arrive.

Malware that encrypts or steals data and demands payment for its return, increasingly paired with the threat of public leaks.

The amount and type of risk an organization is willing to accept in pursuit of its objectives, set by leadership and used to guide decisions.

An access model where permissions attach to named roles, and people get permissions by holding a role, rather than being granted access one item at a time.

A database feature that decides which rows a query is allowed to return based on who is asking, enforced inside the database rather than by application code.

How an organization stores, distributes, rotates, and revokes the credentials its systems need to run: API keys, database passwords, signing keys, and tokens.

Technology adopted by teams without the knowledge or approval of IT and security, from unsanctioned SaaS to personal devices.

Security Information and Event Management: a platform that collects and correlates logs across systems so analysts can detect and investigate incidents.

Security Orchestration, Automation and Response: a platform that runs codified playbooks against a triaged case, executing containment actions across systems and recording the trail.

An independent auditor's report on how a service organization controls customer data, assessed against the AICPA's trust services criteria. Not a certification.

A machine-readable inventory of every component in a piece of software, including dependencies you never chose directly, in a standard format buyers can consume.

Sender Policy Framework: a DNS TXT record listing the servers authorised to send for a domain, checked by the receiver against the connecting server's IP address.

The controls that stop one customer's data in a multi-tenant product from ever reaching another customer, regardless of what the application code does.

Extended Detection and Response: correlates telemetry across one vendor's own sensors, typically endpoint, identity, email and network, and ships its detection content already written.

A security model that treats every request as unauthenticated and unauthorized until proven otherwise, regardless of where it comes from.