Red teaming (AI)

It extends classic red teaming to model-specific failure modes: jailbreaks, prompt injection, data leakage, biased or dangerous outputs, and unsafe tool use. Frontier labs run internal red teams, and regulators increasingly expect testing evidence.

What makes it different from a penetration test is that the target is probabilistic. A vulnerability in conventional software either exists or does not, and a proof of concept reproduces. A model may refuse a request nine times and comply on the tenth, or comply only when the request arrives in the fourth turn of a conversation, or only in one language. Findings are therefore statistical rather than binary, and a single successful attempt is a finding even when the same prompt mostly fails.

That has consequences for how you scope the work. Testing the model alone tells you relatively little about your risk, because most of the exposure lives in the application: what the system is permitted to do, what data the retrieval layer can reach, what happens to output downstream. Red teaming the deployed system, with its real tools and real permissions, surfaces the excessive agency problems that a model-only evaluation never touches.

It also cannot be a one-off. The model changes under you when a provider ships an update, the application changes when someone adds a tool, and the attack literature moves monthly. The practical shape for most organizations is a scoped exercise before launch, automated regression tests for the failure modes you already found, and a re-test when the model version or the tool permissions change. Keep the evidence: it is increasingly what an assessor asks to see.