Question 1 · 4.1
An insurer's claims assistant is ready for a go/no-go review. The team reports “93% accuracy on our test set” and nothing else.
What should the architect ask for before recommending a release?
A A larger test set, so that the accuracy figure is more precise. B Targets and results for latency, cost per claim, safety and security alongside accuracy. C A switch to the most capable model to push accuracy above 95%. D A comparison of the 93% figure with published benchmark scores. Question 2 · 4.1
A retailer's order-change agent succeeds on 80% of trials for a given task. Customers use it once per order and there is no human to pick among attempts.
Which metric best reflects the customer experience across repeated use?
A pass@k, because it shows the agent can solve the task. B Mean tokens per trial, because it tracks efficiency. C ROUGE-L against a reference transcript. D pass^k, because it measures consistent success every time. Question 3 · 4.1
A legal-research assistant's criteria are being drafted. The team wants metrics that would catch real problems before launch.
Which two criteria are well-formed? (Select 2.)
A p95 time to first token under 1.5 seconds in the production UI. B The assistant should be secure against prompt injection. C Average latency should feel fast to lawyers. D Fewer than 1 in 200 red-team documents cause an unrequested tool call. E Accuracy as high as possible within the budget. Question 4 · 4.1
Two models are candidates for a summarisation service. Model A costs less per million tokens than Model B. What is the soundest way to compare cost?
A Pick Model A, because its per-token price is lower. B Run both on the eval set and compare spend per passing summary. C Compare the models’ maximum output token limits. D Estimate cost from the average word count of summaries. Question 5 · 4.2
A team's summarisation eval uses a single LLM judge to score accuracy, tone and completeness in one prompt. Scores look good, but clinicians reviewing outputs keep finding omissions.
What is the best next step?
A Replace the judge with a larger model and keep the single prompt. B Switch entirely to clinician review for every output. C Split into one rubric judge per dimension and measure each against clinician labels. D Raise the pass threshold on the existing combined score. Question 6 · 4.2
An e-commerce team is building the first eval set for a returns assistant. They have two weeks and a backlog of support tickets.
Which two actions give the most useful starting dataset? (Select 2.)
A Turn 20–50 tickets where the old assistant failed into test cases. B Wait until 1,000 cases are ready before running anything. C Include cases where the assistant should decline or escalate. D Use only clean examples written by the product manager. E Grade each case by checking the exact order of tool calls. Question 7 · 4.2
An agent's pass rate on a benchmark-style suite is 40%. Reading transcripts, an engineer sees many “failures” where the answer was correct but formatted slightly differently.
What does this most likely indicate?
A The agent needs a stronger model. B The grader is too rigid and must be fixed first. C The suite has saturated and needs harder tasks. D The results are non-deterministic, so more trials are needed. Question 8 · 4.2
Why should a team keep a held-out evaluation set that is not used during prompt iteration?
A It is required for the API to report token usage correctly. B Held-out cases can be graded without any reference answers. C It lets the team skip regression testing for model upgrades. D Prompts tuned on a set overfit to it; unseen cases give an honest estimate. Question 9 · 4.3
Three days into a planned two-week A/B test, a new support prompt leads the old one by five points on resolution rate. The product manager wants to ship it today.
What should the architect recommend?
A Ship now; a five-point lead is large enough to be real. B Continue for the planned period and apply the pre-set decision rule. C Stop the test and rerun it offline on the eval set instead. D Ship now but keep the old prompt available as a fallback. Question 10 · 4.3
A team plans to move their claims assistant to a newer Claude model and, in the same release, rewrite the system prompt and add a reranker to retrieval.
What is the main problem with this plan?
A Newer models can't be A/B tested against older ones. B Rerankers require a separate evaluation framework. C Prompt rewrites should only be tested in production. D Any change in results can't be attributed to a single cause. Question 11 · 4.3
An internal legal-drafting tool serves about 40 lawyers. The team has a significant prompt redesign ready and wants evidence it's better before rolling it out.
Which two approaches are most appropriate? (Select 2.)
A Run the redesign against the offline capability and regression suites. B Run a two-week A/B test split by request across the 40 users. C Have senior lawyers blind-review paired outputs from both versions. D Ship it and watch for complaints. E Test on five hand-picked contracts. Question 12 · 4.3
A team moves from Claude Sonnet 4.6 to Claude Sonnet 5 because the per-token price is lower. What should they do before full production rollout?
A Re-run the eval suites and re-baseline cost per task on real workload. B Nothing beyond swapping the model ID; it is a drop-in upgrade. C Lower temperature to keep outputs consistent with the old model. D Estimate the saving from the per-token price difference alone. Question 13 · 4.4
A benefits chatbot tells an employee that dental cover includes orthodontics. It doesn't. The transcript shows the retrieved chunks cover general dental benefits but not the exclusions list.
What is the most effective fix?
A Switch to a more capable model so it reasons about exclusions. B Add “never make things up” in capital letters to the system prompt. C Fix retrieval to include exclusions and let the bot say it can't confirm. D Lower the temperature so answers are more deterministic. Question 14 · 4.4
A contract-summary feature intermittently returns summaries that stop mid-sentence. The team suspects the model is “losing focus” on long contracts.
What should be checked first?
A Whether stop_reason is max_tokens on the affected responses. B Whether a larger model summarises long contracts better. C Whether the prompt asks for concise summaries. D Whether the contracts contain prompt-injection attempts. Question 15 · 4.4
A claims-analysis step fails on complex multi-document cases. The team re-runs the failing cases: a clearer prompt with examples fixes almost none; the same prompt on a more capable model fixes most.
Which two conclusions or actions are best supported? (Select 2.)
A The failures are primarily a model mismatch for this slice. B The failures are hallucinations caused by retrieval gaps. C Route complex multi-document cases to the more capable model. D Keep iterating on the prompt until the smaller model passes. E Replace the eval set, since it is too hard for production. Question 16 · 4.4
After a model upgrade, an agent starts calling a search tool on nearly every turn, even for simple questions. Its prompt says “ALWAYS use tools to verify everything.” What is the most likely diagnosis?
A The new model is hallucinating calls to the search tool. B The new model is under-powered for tool selection. C The search tool's description is corrupted. D A prompt failure: wording tuned for the old model now over-triggers. Question 17 · 4.5
A claims assistant sends the same 9,000-token system prompt with every request, about 30 requests a minute. Cost is over budget and quality is fine.
Which change is most likely to cut cost without affecting output quality?
A Move to a smaller model for every request. B Set max_tokens lower to reduce spend. C Cache the system prompt as a stable prefix. D Send requests through the Batch API. Question 18 · 4.5
A team added cache_control to its system prompt, but cache_read_input_tokens is always 0 on Claude Sonnet 5.
Which two causes could explain this? (Select 2.)
A The system prompt starts with the current time, so the prefix differs every request. B The cached system prompt is 700 tokens long. C The responses are longer than 1,000 tokens. D The requests use streaming. E The team uses the default 5-minute lifetime. Question 19 · 4.5
Users of an internal research assistant say it “feels frozen” for several seconds before long answers appear. Total generation time is acceptable to them once text starts.
What is the most appropriate first change?
A Move the workload to the Batch API. B Stream the response so text appears as it's generated. C Lower max_tokens so answers finish sooner. D Switch to the most capable model available. Question 20 · 4.5
A team moves a classification job to a model with a lower per-token price, expecting a 40% saving. The bill falls by far less. What is the most likely explanation?
A Batch discounts don't apply to the new model. B Prompt caching is disabled on newer models. C The new model's tokenizer counts the same text as more tokens. D Output tokens are free on the old model. Question 21 · 4.6
After a prompt update, some users of a report-writing feature say reports end abruptly. The team logs latency and HTTP status for every call, and nothing looks unusual.
Which additional logged field would most directly confirm the cause?
A The stop_reason of each response, grouped by prompt version. B The user's browser and operating system. C The number of retries the SDK performed. D The cache hit rate for the system prompt. Question 22 · 4.6
A finance team wants a daily breakdown of Claude API spend by workspace and model, with an alert when any workspace exceeds its budget.
What is the most appropriate data source?
A Sum usage fields from each application's own logs. B The token-counting endpoint run on each prompt. C The Usage & Cost Admin API. D Claude Code OpenTelemetry metrics for the organisation. Question 23 · 4.6
A team runs an A/B test of two prompts in production and later needs to explain why one variant's cost per resolved ticket rose.
Which two fields must each log line carry for this analysis? (Select 2.)
A The prompt version or variant that served the request. B The token counts from usage, including cache reads. C The full unredacted customer message, kept indefinitely. D Only requests that returned an error. E The monthly invoice total. Question 24 · 4.6
A support assistant's error rate and latency are stable, but the team suspects answer quality has drifted as customer questions changed. What is the best way to monitor this?
A Wait for thumbs-down feedback to rise above a threshold. B Re-run the original offline eval set every night. C Add more latency percentiles to the dashboard. D Sample production transcripts and grade them with calibrated graders.