CCDV-F 7.2 — Guardrails and safe deployment

Guardrails in series around one request

Input guardrailscreen and classify before the call
Constrained promptpolicy, scope, refusal text
Modelproduces text and tool requests
Output guardrailvalidate and screen the reply
Action controlleast privilege, approval, audit log

every layer logged — monitoring closes the loop

Controls sit either side of the model and around its actions. The model's own judgement is one contributor, never the enforcement point.

Content policy: write it down before you enforce it

A content policy is a written list of what your application will not accept or produce, with each category defined well enough that two reviewers would agree. The documented approach to moderation starts here, not with a prompt: enumerate the unsafe categories, then add the definitions and context that make them decidable for your domain — what counts as financial advice on your platform, what counts as harassment in your community.

The moderator itself is then a small, cheap classifier: a fast model, the message wrapped in tags, the categories supplied, and a structured JSON verdict. The guide is explicit that a fast model is the right choice here, and gives the order-of-magnitude cost difference at scale as the reason.

A moderation call with graded outcomespython

PROMPT = """Determine whether this message warrants moderation,
using only the categories below.
<message>{message}</message>
<categories>{categories}</categories>
Return risk 0 (none), 1 (low), 2 (medium) or 3 (high)."""

def moderate(message: str) -> dict:
    r = client.messages.create(
        model=FAST_MODEL, max_tokens=256,
        messages=[{"role": "user",
                   "content": PROMPT.format(message=message, categories=CATS)}],
        output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
    )
    return json.loads(r.content[0].text)

verdict = moderate(text)          # {"risk": 2, "categories": [...], "why": "..."}
if verdict["risk"] == 3:  block(text)            # auto-block
elif verdict["risk"] == 2: queue_for_human(text) # human review
elif verdict["risk"] == 1: warn_user(verdict)    # feedback, allow

Graded risk levels are worth more than a boolean. A binary flag forces one threshold to serve two different mistakes: blocking something harmless, and letting something harmful through. Levels let you auto-block only what you are confident about, send the ambiguous middle to a person, and give feedback on the rest — and they give you something to tune when you measure precision and recall on real traffic.

Layering: what each layer can and cannot catch

Guardrails divide neatly into two families, and a good design uses both. Deterministic controls — schema validation, allowlists, regular expressions, permission checks, rate limits — are cheap, fast and certain, but only catch what you can specify in advance. Model-based controls — a classifier screening input, a reviewer screening output — catch meaning rather than strings, but they are probabilistic and cost a call. Put the deterministic ones first: there is no sense paying a model to notice that a field is missing.

Layer	Catches	Misses
Input classifier	Harmful or manipulative requests, injection attempts	Novel phrasings; anything arriving later via tools
System prompt policy	Ordinary drift; sets refusal behaviour	A determined attacker; it is guidance, not enforcement
Schema validation of output	Malformed or out-of-range results	Well-formed answers that are simply wrong
Output classifier	Leaked secrets, policy-breaking content	Subtle errors; adds latency to every reply
Permission and scope checks	Actions the caller may not take, whatever the model asked	Nothing in this row — this is the enforcement layer
Human approval	Anything irreversible or high-value	Volume; people cannot review everything

Worked example · Moderating a marketplace's listings

A second-hand marketplace lets sellers write free-text listings. The team wants to catch prohibited items and abusive descriptions before publication, across six languages, without adding noticeable delay to posting.

Write the policy first. Eleven categories, each with two or three lines of definition and a borderline example. The document, not the prompt, is the source of truth.
Classify with a fast model. Each listing goes to a small model with a structured schema returning a risk level and the categories hit. Latency is well inside the posting flow's budget.
Grade the response. Level 3 blocks with an explanation to the seller; level 2 publishes nothing and queues for a human; level 1 publishes with a nudge; level 0 publishes.
Measure before trusting. Two thousand historical listings, hand-labelled, give precision and recall per category. Two categories are too eager and their definitions are tightened.
Fail closed on the risky path. If the classifier errors, the listing queues for human review rather than publishing — a small backlog, never a silent bypass.
Keep watching. Category counts are tracked weekly, so a new evasion pattern shows as a shift in the numbers rather than as a headline.

Takeaway: The moderation model was the least interesting part. The policy document, the graded thresholds, the labelled evaluation set and the failure behaviour are what made it safe to switch on.

Secure by design: least privilege and identity

Secure by design means the safe configuration is the default one, so a mistake produces a refusal rather than an incident. Claude Code is a useful worked example because its model is documented in detail: in its manual mode it starts read-only and asks before editing files or running commands that modify the system, while a built-in set of read-only commands runs without a prompt. Writes are confined to the directory it was started in and its subdirectories. Bash commands can be sandboxed with filesystem and network isolation. First use of a codebase or a new MCP server requires trust verification, and organisations can impose managed settings that a user cannot override.

Two ways to give an agent access

Convenient

One credential with broad rights, shared by all features
Every tool available on every request
Writes anywhere on the filesystem
Approval prompts disabled because they were annoying
Actions logged only when they fail

Least privilege

A scoped credential per tool, read-only where possible
Only the tools this task needs, attached per session
A working directory boundary, sandboxed execution
Human approval on irreversible or high-value actions
Every tool call logged with the caller's identity

Both agents can do the day-to-day work. Only one of them still does something small when an injected instruction arrives.

Identity and access management is the same discipline you would apply to any service, with one extra rule: the agent acts for a person, so it must act as that person. The identity comes from the authenticated session and is passed to the tool layer by your code; it is never an argument the model fills in. An agent that holds its own super-user credential has, in effect, granted every user the union of everyone's permissions.

Privacy belongs in the same design pass rather than in a later review. Minimise what enters the context, redact identifiers the task does not need, choose the retention posture your obligations require, and keep an audit trail of who asked for what. Those choices are cheap before launch and expensive afterwards; the detail of retention options and PII handling is in 7.1, and credential mechanics are 7.4.

A pre-deployment guardrail review

Passes: Written content policy with defined categoriesDecidable by two reviewers, not just by the model
Passes: Input screening before the main callFast model, structured verdict
Passes: Output validated against a schema before useDeterministic and cheap; do it first
Check: Tools scoped to the task, credentials read-only where possibleLeast privilege is the enforcement layer
Check: Human approval on irreversible actionsDeletion, payment, external send
Missing: Defined behaviour when a guardrail errorsFail closed on high-risk paths, and test it
Missing: Audit log of prompts, tool calls and decisionsWithout it you cannot investigate or improve

Run this before launch, not after the first incident. The two “missing” rows are the ones teams most often discover the hard way.

Worked example · Rolling out an internal operations agent

An operations team wants an agent that can read tickets, query the warehouse database and update stock records. The first design gives it a database account with full read and write access to every table, because that was the account the team already had.

Separate the capabilities. Reading tickets, querying stock and writing stock are three different rights, and almost every request needs only the first two.
Scope the credentials. A read-only account for queries; a second account whose write access is limited to the stock table, used only by the update tool.
Bind to the caller. The tool layer takes the operator's identity from the session and records it on every write, so the audit trail names a person rather than the agent.
Gate the irreversible step. Stock adjustments above a threshold produce a proposed change that an operator confirms, rather than an immediate write.
Stage the rollout. Read-only for two weeks with the write tool disabled, while the team reads the logs and compares proposed changes against what operators would have done.
Enforce it centrally. The permission configuration ships as managed settings so an individual cannot widen it locally.

Takeaway: Nothing here is specific to AI. It is the permission review any new service would get — which is exactly the point, and exactly the step that gets skipped when the new service is called an assistant.

Traps the wrong answers are built from

Tempting but wrong	Do this instead
Treating a system-prompt policy as the guardrail	Enforce in code — validation, permissions, approvals — and keep the prompt as one layer.
A moderation call whose errors are treated as “no violation”	Decide the failure mode explicitly and fail closed on high-risk paths.
A single binary block/allow threshold	Grade risk so the ambiguous middle reaches a human instead of being guessed at.
Giving the agent one broad credential for every capability	Scope credentials per tool, read-only by default, bound to the caller's identity.
Shipping guardrails without measuring them	Label real traffic and track precision and recall per category before and after launch.
Leaving permissions to individual configuration	Enforce organisational standards centrally with managed settings.

You should now be able to

Write a content policy with decidable categories and graded risk levels.
Build a fast, structured moderation classifier and route its verdicts appropriately.
Layer deterministic and model-based guardrails, and say what each layer cannot catch.
Define fail-closed behaviour for a guardrail that errors, and test it.
Apply least privilege and session-bound identity to an agent's tools and credentials.
Run a pre-deployment guardrail review covering policy, screening, permissions, approval and audit.

Guardrails and safe deployment