Guardrails and safety controls

What a guardrail is — and where it lives

A guardrail is any control that keeps a system inside the behaviour you intended. Some live inside the model’s context (the system prompt, examples, a permission to say “I don’t know”). Others live in your code and infrastructure (a classifier that screens input, a deny rule that removes a tool, a regex over the output, a rate limit on an abusive account). The architect’s job is to place each control where the risk enters, and to prefer deterministic controls wherever the cost of a miss is high.

Anthropic’s guardrail guidance groups the work by threat. Against direct jailbreaks, it suggests a lightweight harmlessness screen before the main call, input validation against known patterns, a system prompt that states boundaries and how to decline, and monitoring that throttles or bans repeat offenders. Against indirect prompt injection — instructions hidden inside a web page, email or uploaded file — it suggests delivering third-party content only inside tool_result blocks, labelling where it came from, telling Claude in the system prompt that such content is data and never a command, wrapping it in JSON so an attacker cannot break out of the delimiters, and screening each tool’s raw output with a small classifier before it reaches the main model.

Input and output screens

A screen is a separate, cheap check that runs before or after the main call. Anthropic’s docs describe using a small, fast model such as Haiku to classify input as harmful or not, returning a structured yes/no so your code can branch on it reliably. The same idea works on the way out and on tool output: a second model, a keyword list or a regex looks for leaked secrets, policy breaches or signs that an injection succeeded. Anthropic’s “Building effective agents” describes this as parallelisation used for guardrails — one model instance handles the request while another screens it.

Screens have costs: added latency, added spend, and false positives that block legitimate users. That is why the prompt-leak guidance tells you to use leak-resistant techniques only when truly needed and to test that they do not degrade the main task, and suggests trying monitoring and output post-processing first. A good design sizes the screen to the risk: a public chatbot for a bank earns an input screen; an internal summariser over trusted reports may not.

Least privilege for agents

Agents raise the stakes because the model’s output becomes an action. The jailbreak guidance says not to give Claude secrets it does not need, to run tools in sandboxes, and to scope permissions as narrowly as possible. The Claude Agent SDK turns this into configuration. Permission checks run in a fixed order — hooks, deny rules, ask rules, the permission mode, allow rules, then your canUseTool callback. A deny rule blocks a tool even in bypassPermissions mode, and a bare deny such as Bash removes the tool from Claude’s context entirely.

const options = {
  // Pre-approve only read-only tools.
  allowedTools: ["Read", "Glob", "Grep"],
  // Remove the shell entirely: Claude never sees it.
  disallowedTools: ["Bash"],
  // Anything that would need a prompt is denied, not asked.
  permissionMode: "dontAsk",
};

One detail is worth remembering for trade-off questions: the SDK docs warn that allowedTools does not constrain bypassPermissions. Listing a few tools as allowed while running in bypass mode still approves every other tool. If you need a hard boundary, use deny rules or a PreToolUse hook, which the docs describe as the place for checks that must run on every call.

Guardrails against wrong answers, not only bad actors

Not every guardrail is about attackers. The hallucination guidance lists controls that keep an honest model honest: explicitly allow Claude to say it does not know, ask it to extract direct quotes before reasoning over long documents, require a citation for each claim and retract claims with no supporting quote, and restrict it to the provided documents when general knowledge is not wanted. The docs are candid that these reduce hallucinations but do not eliminate them, so high-stakes outputs still need validation — which is where human review (5.3) comes in. Diagnosing a hallucination after launch is covered in 4.4.

Traps the wrong answers are built from

You should now be able to

• Place guardrails in layers — input, prompt, isolation, permissions, output, monitoring — according to where each risk enters.
• Distinguish direct jailbreaks from indirect prompt injection and choose the controls that fit each.
• Apply least privilege to agent tools using deny rules, permission modes and hooks, and explain why prompt instructions alone are insufficient.
• Justify when a classifier screen is worth its latency, cost and false-positive rate.
• Use grounding controls — permission to abstain, quotes, citations — as guardrails against hallucination.