Eval, Testing, and Debugging

Question 1 · 4.1

An invoice-extraction service returns JSON that is occasionally cut off mid-object. The API call returns HTTP 200, and the response has a stop_reason of max_tokens. The team has wrapped the parse in a retry with exponential backoff.

What is wrong with the current handling?

• ABackoff is too short; the service needs a longer delay between attempts.
• BThe response is truncated, not failed, so it should be continued or given a higher max_tokens.
• CThe model is producing invalid JSON and the schema should be simplified.
• DThe request exceeded the context window and older messages must be trimmed.

Question 2 · 4.1

An agent in production began giving confident but wrong summaries after a deployment. The prompt did not change. A trace shows several turns where an assistant message requesting two tools is followed by a user turn with one result block.

Where does the fault lie, and what should be done first?

• AThe model; add a stronger instruction about not guessing.
• BThe model; switch to a more capable model for these summaries.
• CThe integration layer; ensure each tool_use gets a matching tool_result.
• DThe tools; add retries inside each tool before returning.

Question 3 · 4.1

A scheduled Claude Code job has started taking far longer than usual. Exit codes are zero, the output is correct, and nothing appears in the application error log.

What is the most useful next step?

• ARaise the job's turn limit so it has more room to finish.
• BInspect the run's retry events in stream-json output.
• CSwitch the job to a smaller model to reduce latency.
• DAdd a timeout so that slow runs fail fast and raise an alert.