CCAR-P 6.3 — Feedback loops, expectations and SLAs

The stakeholder feedback loop

MeasureSLIs, eval scores, user ratings
Review with clientfixed cadence, same scorecard
Triage feedbackbug, gap, new scope, or noise
Change & re-testagainst the regression set

Every sprint or month → measure again and report the trend

A loop with a fixed rhythm. Feedback is collected and triaged against the agreed criteria before anything is changed, and every change is re-tested.

SLIs, SLOs and SLAs — three different things

Google’s SRE book gives the standard vocabulary. A service level indicator (SLI) is a measured number, such as request latency, error rate or availability. A service level objective (SLO) is a target for that number, set by the team that runs the service. A service level agreement (SLA) is a contract with the customer that states consequences — refunds, credits, penalties — if objectives are missed; business and legal teams negotiate it, and engineers advise on what is feasible. The book notes that people often say “SLA” when they mean SLO. On the exam, and with clients, keep them apart.

From measurement to contract

SLAcontract: target + consequences
SLOinternal target, tighter than the SLA
SLIwhat you actually measure

Each layer builds on the one below. You cannot sign an SLA on a number you are not measuring.

The SRE guidance adds several habits worth copying. Choose a handful of indicators that reflect what users care about, not everything you can measure. Use percentiles rather than averages, because an average latency hides the slow tail that angry users experience. Keep your internal SLO tighter than what you promise externally, so you have a safety margin. And do not wildly overachieve: users come to rely on the performance you actually deliver, not the one you wrote down.

SLAs for AI systems: speed, uptime and quality

Traditional SLAs cover availability and latency. A Claude-based system also needs agreed quality targets, because a fast, available system that gives wrong answers is still failing. Anthropic’s documentation on success criteria lists dimensions like task fidelity, consistency, tone, privacy preservation, latency and price. Quality targets usually belong in the SLO and the regular review, measured on a maintained test set, rather than in a financial-penalty SLA — model output is probabilistic, and a contract should promise the process (measurement, review, remediation time) as well as the number.

Area	Example SLI	Example SLO	Commit externally?
Availability	Successful requests ÷ total	99.5% monthly	Only if your dependencies support it
Latency	95th-percentile time to first token	< 3 s in business hours	Yes, with a margin
Quality	Score on the 300-case test set	≥ 94% each monthly run	As a reviewed target, not a penalty
Escalation	Share of chats handed to a human	≤ 15%	As a reported metric
Support	Time to acknowledge a P1 incident	30 minutes	Yes

Two Claude platform facts matter when you set expectations. First, capacity: the rate limits page describes requests-per-minute and input- and output-tokens-per-minute limits per model, usage tiers that rise with an organisation’s history, and a way to request increases. Cached input tokens do not count towards the input-token limit on most models, so prompt caching raises effective throughput. Second, service tiers: the docs describe a Priority Tier with a 99.5% uptime target for organisations that hold capacity commitments, the default standard tier, and batch processing for work that can wait.

Worked example · A BPO’s contact-centre assistant

A business-process outsourcer is deploying a Claude assistant that suggests replies to agents handling a telecom client’s customers. The telecom’s contract with the BPO has strict response-time terms, and the account director wants to promise “99.99% uptime and instant answers”.

Check the dependencies. The assistant depends on the Claude API (standard tier), the BPO’s knowledge base and the telecom’s CRM. None of them publishes a 99.99% commitment to the BPO.
Measure first. A four-week pilot records 95th-percentile suggestion latency of 2.4 s and 99.7% successful requests, with most failures being brief rate-limit errors at peak.
Design for the gap. You add retries honouring retry-after, prompt caching on the long shared system prompt, and a graceful fallback: if no suggestion arrives in 5 s, the agent simply answers unaided — the call never waits on the AI.
Propose the targets. Internal SLO: 99.7% of suggestions within 4 s. External commitment: 99.5% within 5 s, with the fallback guaranteeing no customer-facing outage. Quality is reported monthly against the test set.
Agree the review. A monthly service review with the account director and the telecom’s operations lead uses the same scorecard every time.

Takeaway: The promise got smaller and more believable. Because the fallback keeps calls flowing, the outcome the telecom cares about is protected even when the AI is slow.

Running the feedback loop

Expectation alignment is not a single conversation. Set a cadence at the start — a weekly check-in during a pilot, a monthly service review after launch — and use the same scorecard each time so trends are visible. Collect feedback from several channels: user thumbs-up/down, comments from reviewers, support tickets, and the regular eval run. Then triage it. Some feedback is a bug; some reveals a gap in the test set; some is new scope that needs a change request; some is one person’s preference.

Triage before you act

What kind of feedback is this?

System misses an agreed criterion
Defectfix, add case to test set
Failure the tests never covered
Test gapadd cases, re-baseline
Asks for new behaviour
New scopechange request, re-estimate
Single preference, no pattern
Log & watchact if it recurs

Every change goes through the same regression set, so fixing one complaint does not quietly break something else.

Worked example · A bank’s “the bot got worse” escalation

Three weeks after launch, a retail bank’s head of digital emails that the mortgage FAQ assistant “has got noticeably worse” and asks for the prompt to be rewritten today.

Check the scorecard. The weekly eval run is steady at 95% against the target of 93%. Latency and escalations are unchanged.
Look at the evidence. The complaints all concern a new fixed-rate product launched last week, which is not in the knowledge base or the test set.
Classify it. This is a content gap plus a test gap, not a model regression. You add the product documents, write 20 test cases for them, and re-run the full suite.
Close the loop. You reply the same day with the diagnosis, the fix date and a new process step: product launches now trigger a knowledge-base and test-set update.

Takeaway: Evidence turned an emotional escalation into a specific fix — and a process change that stops it recurring. A same-day prompt rewrite could have broken the 95% that was working.

Traps the wrong answers are built from

Tempting but wrong	Do this instead
Promising an uptime figure higher than the model API and other dependencies support.	Derive commitments from measured SLIs and dependency terms, with fallbacks and a safety margin.
Using “SLA” for every target.	Separate SLIs (measurements), SLOs (internal targets) and SLAs (contracts with consequences).
Reporting average latency.	Report percentiles such as the 95th or 99th so the slow tail is visible.
Changing prompts immediately in response to the loudest complaint.	Triage against the scorecard, then change and re-test against the regression set.
Only reviewing the system when something goes wrong.	Hold a fixed-cadence review with the same scorecard so trends are visible.

You should now be able to

Distinguish SLIs, SLOs and SLAs and say who owns each.
Set service levels from measured data, keeping internal SLOs tighter than external commitments.
Account for Claude API rate limits and service tiers when promising availability and throughput.
Include quality targets measured on a test set alongside latency and availability.
Run a regular feedback loop that triages input into defects, test gaps, new scope and noise.

Feedback loops, expectations and SLAs