CCAR-F 4.1 — Explicit criteria for precise prompts

Vague instruction versus explicit criteria

Vague

Review this pull request.
Flag any problems you see.
Be conservative and only
report important issues.

Explicit criteria

Report ONLY these as bugs:
- logic that returns wrong
  results for a valid input
- queries not scoped to
  the caller's tenant
- PII written to logs
Do NOT report: style, naming,
anything the linter checks.
Each finding needs a file:line
and the input that breaks it.
If unsure, mark it “unverified”.

The left prompt leaves Claude to invent the threshold. The right one says what to report, what to skip, how severe each kind is, and what evidence a finding needs.

Why vague prompts produce false positives

A false positive is a finding that should not have been raised: a “bug” that is not a bug, a support ticket tagged urgent that is not, a contract clause flagged as risky that is standard. They happen when the prompt names a goal (“find problems”, “flag risky clauses”) but not a boundary. Claude then has to guess the boundary, and its guess will not match yours — nor stay the same across thousands of inputs.

Anthropic’s prompting guide puts the fix plainly: Claude responds well to clear, explicit instructions, and its golden rule is to imagine handing your prompt to a colleague with no context. If they would have to ask “what counts as important?”, so does the model. The guide also notes that recent Claude models follow instructions more literally than older ones. That cuts both ways: a precise criterion is applied precisely, and a vague one is applied vaguely.

False positives are not just wasted effort. When one automated reviewer comment in three is noise, developers learn to skim past all of them — including the accurate ones. Precision is what keeps the channel worth reading, which is why the CI scenario in the exam guide is framed around minimising false positives on pull requests.

What an explicit criterion contains

An explicit criterion is something two reviewers could apply to the same input and agree on. It usually has five parts. Not every prompt needs all five, but a noisy one is almost always missing at least two.

Checking a review prompt against the five parts

Check: Report when — concrete categories“any problems” names no category
Missing: Do not report — the skip liststyle and lint noise gets flagged
Missing: Severity defined with examples“important” means something different each run
Missing: Evidence bar for each findingguesses from function names get posted
Fails: What to do when uncertain“be conservative” hides doubt instead of labelling it

A typical first-draft review prompt. It has a goal and nothing else — each missing part is a place the model fills in its own judgement.

Part	Vague	Explicit
Report when	“Flag security issues”	“Flag SQL built by string concatenation from request data”
Do not report	(nothing said)	“Skip formatting, naming, generated files under `src/gen/`”
Severity	“Mark serious ones”	“Important = breaks behaviour, leaks data or blocks rollback; everything else is Nit at most”
Evidence	(nothing said)	“Cite `file:line` and describe an input that triggers it”
Uncertainty	“Only report if sure”	“If you cannot show a triggering input, label it unverified”

Claude Code’s managed Code Review feature is a useful real-world model of this. Teams tune it with a REVIEW.md file, and the documentation’s advice maps onto the five parts almost exactly: redefine what Important means for your repository, list paths and categories where it should post nothing, and set a verification bar such as requiring a file:line citation rather than an inference from naming. It also warns that a long file dilutes the rules that matter — criteria work because they are few and sharp.

A classification prompt with explicit criteriatext

<task>Classify each support ticket's urgency.</task>

<criteria>
urgent  — the customer cannot use a paid feature right now
          (outage, locked account, failed payment on renewal day)
normal  — something is wrong but there is a workaround
low     — questions, feature requests, feedback
</criteria>

<not_urgent>
Angry tone, capital letters or "ASAP" do not make a ticket urgent
by themselves. Classify by impact, not by wording.
</not_urgent>

<why>Urgent tickets page an on-call engineer at night, so a false
"urgent" costs someone their sleep.</why>

If the ticket does not say whether a workaround exists, answer
"normal" and set needs_review to true.

Notice the <why> block. The prompting guide recommends explaining the reason behind an instruction because Claude generalises from the explanation. “Never mark angry tickets urgent” is a rule the model can apply mechanically; “urgent pages a human at night” lets it reason about cases your list did not foresee.

Worked example · A bank’s pull-request reviewer nobody reads

A bank’s platform team runs Claude on every pull request. The prompt says “Review for bugs, security issues and code quality. Be thorough.” After a month, a sample of 200 comments shows 45% are style suggestions or speculative warnings, and engineers have started resolving them unread.

Measure first. The team labels the 200 comments: real bug, real but minor, false positive. Most false positives fall into three groups — naming and formatting, “might be null” guesses on values the type system already guarantees, and warnings in generated client code.
Write a criterion per group. Style and naming go on the do-not-report list (the linter already enforces them). Null warnings need a traced path showing where the null comes from. src/generated/ is skipped entirely.
Define severity. Important is reserved for wrong results, unscoped account queries, and secrets or card numbers in logs. Everything else is a Nit, capped at five per review.
Add an evidence bar. Each Important finding must cite file:line and describe the input that triggers it.
Re-run on the same 200-PR sample. Comment volume drops sharply; the labelled real bugs are still found, and the ones that were missed tell the team which criterion to widen next.

Takeaway: The team did not ask the model to try harder or be more careful. It told the model, category by category, where the line was — and checked the result against the same labelled sample.

“Be conservative” is not a criterion

The most tempting fix for false positives is a sentence like “only report issues you are highly confident about”. It feels like it should raise precision. In practice it moves the threshold without saying where to, and it silently cuts recall: real findings disappear along with the noise, and nothing in the output tells you which.

The same logic applies to shouting. The prompting guide notes that newer models are more responsive to the system prompt and can overtrigger on aggressive language like “CRITICAL: you MUST”. Capital letters make an instruction louder, not clearer. If a rule is being missed, make it more specific or explain why it matters; do not make it louder.

Tightening a noisy prompt

Label a samplereal, minor or false positive
Group the noisewhich kinds of wrong flags recur?
Write a criterionskip rule, severity or evidence bar
Re-run the setprecision up, recall unchanged?

repeat for the next largest group of false positives

Criteria come from the false positives you actually see, and each change is checked against the same labelled set so you know you did not lose real findings.

Anthropic’s evaluation guide makes the same point about success criteria: “good performance” is not a target, “accurate sentiment classification” measured on a test set is. Treat the prompt’s criteria and your evaluation’s criteria as one thing. If you cannot say how you would grade an output as a false positive, you have not yet written the criterion that prevents it.

Worked example · Flagging risky clauses at a law firm

A law firm uses Claude to flag “unusual or risky clauses” in supplier contracts before a lawyer reads them. Lawyers complain that every limitation-of-liability clause gets flagged, even the firm’s own standard wording.

The prompt has a goal (risky) but no reference point. Standard clauses look risky in the abstract.
The firm adds its standard clause library in a <standard_clauses> block and a criterion: flag a clause only if it departs from the matching standard clause in a named way — a cap below the contract value, a missing carve-out for data breaches, an indemnity that runs one way only.
It adds an evidence rule: quote the clause, quote the standard wording it departs from, and name the difference. Asking for quotes first is also one of Anthropic’s documented ways to reduce hallucinated findings.
Clauses that do not match any standard clause are reported in a separate “no reference available” list, rather than being forced into risky or safe.

Takeaway: “Risky” became “departs from our standard in one of these named ways, shown by quotation”. The unknowns got their own bucket instead of inflating the false-positive count.

Where criteria stop and other controls start

Explicit criteria improve how consistently the model draws the line, but a prompt is still probabilistic. When a rule must hold every time — never post a finding without a file:line, never mark more than five nits — enforce it in code after the model responds, using a structured output with a severity field (see 4.3) and validation around it (4.4). The prompt defines the categories; the schema and code enforce them. Showing the model worked examples of borderline cases is the other big lever, covered in 4.2.

Traps the wrong answers are built from

Tempting but wrong	Do this instead
Telling the model to “be conservative” or “only report high-confidence issues”	Define which categories to report and skip; filter by explicit severity after the fact.
Adding CRITICAL / MUST in capitals to a rule that is being missed	Make the rule more specific and explain why it matters.
Naming a goal (“flag risky clauses”) with no reference point	Give the standard or baseline, and name the departures that count.
Letting one noisy category run on and on	Narrow it with a criterion, or switch it off until you can; noise erodes trust in the accurate findings.
Judging a prompt change by how the output looks	Re-run a labelled sample and check precision and recall together.

You should now be able to

Rewrite a vague review or classification instruction as concrete report and skip criteria.
Define severity levels with examples so they are applied the same way on every input.
Add an evidence requirement that filters speculative findings.
Explain why “be conservative” and aggressive emphasis lower recall without defining precision.
Use a labelled sample to find recurring false-positive groups and write a criterion for each.

Explicit criteria for precise prompts