CCAR-P 5.5 — Bias, fairness and transparency

Three questions, three kinds of evidence

Ethical AI reviewfor systems that affect people

Biasdoes the output shift by group? → test
Fairnesswhat outcome is fair here? → decide
Transparencywho knows what AI did? → disclose
Accountabilitywho owns the decision? → name

Bias is measured, fairness is decided and documented, transparency is delivered to the people affected. Each needs its own artefact.

Bias: a known property you must measure

Language models learn from human-written text, so they can reproduce patterns in it. Anthropic studied this directly: researchers generated prompts for 70 decision scenarios — the kinds of yes/no decisions a business or government might make about a person — and systematically varied demographic details within each one. Without interventions, they found patterns of both positive and negative discrimination in select settings. They also found that careful prompt engineering could significantly reduce both. And they were explicit that they do not endorse or permit using language models to make automated decisions for the high-risk use cases they studied.

Two lessons follow for architects. First, bias is not hypothetical, and it can run in either direction — favouring a group is also a distortion. Second, mitigations work but must be verified: a prompt that says “be fair” is a hypothesis until you have measured its effect on your own cases.

Counterfactual testing

Real case setrepresentative, reviewed inputs
Make variantschange only name, age, gender…
Run each many timessame prompt, same model
Compare outcomesgaps by group, with counts

Gap found → change prompt or design, then re-test

Hold everything constant except one attribute. Any systematic gap between the paired results is evidence of bias that the design must address.

A minimal counterfactual check (sketch)python

import itertools

NAMES = {"group_a": ["Emily Walsh"], "group_b": ["Lakisha Brown"]}
AGES = [29, 61]

def decide(case_text: str) -> str:
    r = client.messages.create(
        model=MODEL, max_tokens=20,
        messages=[{"role": "user", "content": PROMPT + case_text}],
    )
    return r.content[0].text.strip()   # e.g. "YES" or "NO"

results = []
for case in CASES:                     # real, reviewed application texts
    for (group, names), age in itertools.product(NAMES.items(), AGES):
        for name in names:
            variant = case.format(name=name, age=age)
            for _ in range(5):         # repeat: outputs can vary
                results.append((group, age, decide(variant)))

# Then compare approval rates per group and age, with counts.

Fairness: a decision you make and document

Bias is something you measure; fairness is something you define for the use case. There is no single technical definition that fits everything. For a loan pre-screen, fairness may mean that equally qualified applicants get equal recommendations regardless of group. For a patient-letter generator, it may mean that reading level and tone are equally clear for every patient. For a retailer’s promotions, it may mean no group is systematically shown worse prices. The architect’s job is to get the business owner to state the definition, turn it into measurable checks, and record the decision and its trade-offs (communicating trade-offs is covered in 6.2).

Use case	What “fair” might mean here	How to check it
Loan pre-screen	Equal recommendations for equally qualified applicants	Counterfactual pairs; approval rates by group
CV screening	Skills drive the shortlist, not names or gaps	Swapped-name variants; reviewer audit
Patient letters	Equally clear and respectful for all patients	Readability and tone checks across groups
Benefits enquiries	Same accuracy for every language and region	Evaluation set sliced by language and region

Worked example · A regional bank’s small-business loan pre-screen

A bank wants Claude to read small-business loan applications and recommend “proceed” or “decline” to an underwriter. The head of lending asks: “How do we know it is fair?”

Define fairness. The bank agrees: applicants with the same financials and business plan should get the same recommendation regardless of the owner’s name, gender or age.
Build the test set. 300 past applications, reviewed by underwriters, each with counterfactual variants that change only the owner’s personal details.
Measure. The first run shows a small but consistent gap for older applicants in borderline cases.
Mitigate and re-test. The prompt is revised to base the recommendation only on listed financial criteria and to cite which ones drove it; the gap closes on re-test.
Keep people accountable. Lending decisions are a high-risk use case under the Usage Policy, so an underwriter decides every application and the test re-runs whenever the prompt or model changes.

Takeaway: Fairness became a written definition, a test set, a measured result and an owner — things an auditor can inspect.

Transparency: who is told what

Transparency operates at three levels. Disclosure tells people an AI is involved: Anthropic’s Usage Policy requires consumer-facing chatbots and agents to disclose that users are interacting with AI rather than a human, at a minimum at the start of each chat session, and for high-risk use cases requires disclosure that AI assisted with the output. Explanation tells people how an outcome was reached: GDPR requires meaningful information about the logic involved in automated decision-making covered by Article 22, and gives people ways to contest. Traceability lets reviewers and auditors reconstruct what happened: which inputs, which prompt version, which sources, which human signed off.

Traceability is where design choices pay off. Asking Claude to cite the source passage behind each claim — a technique from the hallucination guidance — doubles as an explanation mechanism, because every statement points back to evidence a person can read. Logging the prompt version and model alongside each decision means a complaint months later can be investigated rather than guessed at.

Ethics review of a benefits-letter assistant

Fails: Letters say AI assisted in draftingno disclosure
Passes: Each statement cites the case record
Missing: Accuracy tested per language offered
Passes: Caseworker approves every letter
Check: Citizens told how to contestburied in footer
Missing: Prompt and model version logged

The model behaves well in testing, but three items fail on transparency and testing. Those are design gaps, not model gaps.

Traps the wrong answers are built from

Tempting but wrong	Do this instead
Removing protected attributes and declaring the system unbiased.	Test outcomes with counterfactual variants; proxies such as names and postcodes carry the same signal.
Adding “be fair and unbiased” to the prompt and moving on.	Treat prompt mitigations as hypotheses and measure their effect on your own case set.
Letting the model make automated high-risk decisions because tests looked fair.	Keep a qualified human accountable; Anthropic does not endorse automated decisions in these use cases.
Hiding the AI’s role to make the experience feel more personal.	Disclose AI involvement at the start of sessions and in high-risk outputs, as the Usage Policy requires.
Measuring only average accuracy.	Slice results by group, language and region so a gap is not hidden by a good average.

You should now be able to

Explain how bias arises in language models and why it can favour as well as disfavour groups.
Design counterfactual tests that detect bias in decisions about people.
Work with business owners to define fairness for a use case and turn it into measurable checks.
Apply disclosure, explanation and traceability requirements, including Usage Policy disclosure rules and GDPR transparency obligations.
Integrate fairness checks into the evaluation suite so changes are re-tested.

Bias, fairness and transparency