Where LLM systems fail
- Confident errorshallucinated facts, citations
- Manipulationjailbreaks, prompt injection
- Knowledge limitstraining cutoff, missing context
- Inconsistencydifferent runs, different answers
- Compoundingagent errors build on errors
- Human over-trustreviewers stop checking
Limitations versus failure modes
It helps to separate two ideas. A limitation is a property of language models that no prompt removes: they generate plausible text rather than looking facts up, they only know what was in training data or what you put in the context, and their output can vary between runs. A failure mode is the way a particular system breaks when a limitation meets a design choice: a support bot that invents a refund policy, an agent that deletes the wrong records, a summary that silently drops the one sentence that mattered. Architects cannot remove limitations, but they choose which failure modes the system is exposed to.
| Risk | What it looks like | Typical design response |
|---|---|---|
| Hallucination | Fluent, confident, unsupported claims | Ground in sources, require citations, allow “I don’t know” |
| Prompt injection | Hidden instructions in documents or web pages | Isolate untrusted content, least-privilege tools (5.1) |
| Stale or missing knowledge | Answers reflect training data, not today | Retrieve current data; state the source and date |
| Inconsistency | Same question, different answers | Structured outputs, evals over many runs, compare samples |
| Compounding in agents | An early mistake drives later actions | Checkpoints, stopping conditions, sandboxed testing |
| Prompt or data leakage | Instructions or sensitive data appear in output | Keep secrets out of context, post-process output |
| Bias | Different outcomes for different groups | Counterfactual testing, human review of decisions (5.5) |
| Automation bias | Reviewers approve without reading | Review design that forces evidence checks (5.3) |
Confident errors and why they are hard to see
Anthropic’s hallucination guidance is frank that its techniques — letting Claude admit uncertainty, extracting quotes first, citing each claim, checking consistency across several runs — reduce hallucinations without eliminating them, and that critical information still needs validation. The architectural risk is not only that errors occur, but that they look exactly like correct answers. A wrong figure in a well-formatted table carries the same authority as a right one. That is why detection mechanisms such as citations matter: they turn an invisible error into a checkable claim.
Inconsistency is related. The docs suggest “best-of-N” comparison — running the same prompt several times — precisely because disagreement between runs is a signal of an unreliable answer. For a risk review, that means a single successful demo proves little; evaluate across many inputs and many runs before believing a system is dependable.
Failure modes that architecture creates
Connecting a model to tools, documents and other agents adds failure modes the model alone does not have. Anthropic’s “Building effective agents” notes that the autonomy of agents brings higher costs and the potential for compounding errors, and recommends extensive testing in sandboxed environments with appropriate guardrails, plus stopping conditions such as a maximum number of iterations. The jailbreak guidance adds that any content an agent reads from outside — an email, a web page, text extracted from an image — can carry instructions, so every new data source is also a new attack surface.
How one error compounds in an agent
- Misreads inputwrong customer ID extracted
- Queries wrong recordtool returns plausible data
- Acts on itsends letter, updates account
- Reports successsummary looks correct
Running a risk review
A practical risk review takes each part of the design and asks the same questions: what can go wrong here, how bad is it, would anyone notice, and what control applies? It produces a risk register with an owner for each item. Diagnosing a live incident is a different skill (4.4); here you are anticipating failure before launch so the design can absorb it.
Risk review of the claims-intake design
- Passes: System prompt states scope and refusal rules
- Fails: Email content isolated as untrusted datapasted into the prompt
- Missing: Extracted policy number verified against sender
- Missing: Customer-facing decisions reviewed by a handler
- Passes: Iteration and retry limits set
- Check: Evaluated across many runs, not one demo20 cases only
Some risks are about people and data rather than the model. Bias is a known limitation: Anthropic’s own study of model decisions across 70 scenarios found patterns of both positive and negative discrimination without interventions, and the researchers state they do not endorse using models for automated high-risk decisions (5.5 goes deeper). Leakage is another: a prompt containing internal pricing formulas can reveal them, which is why the prompt-leak guidance starts with not putting unnecessary sensitive detail in the prompt at all.
Traps the wrong answers are built from
| Tempting but wrong | Do this instead |
|---|---|
| Listing “hallucination” as the only risk of an LLM system. | Walk every part of the design — inputs, tools, actions, people — and name specific failure modes. |
| Treating one successful demo as evidence of reliability. | Evaluate over many inputs and repeated runs; inconsistency between runs is itself a warning sign. |
| Reviewing only the final output of an agent. | Add checkpoints and verification between steps, since early errors compound into confident final results. |
| Assuming the model knows current facts or internal data. | Retrieve current and internal data explicitly and show where each answer came from. |
| Ranking risks by how likely they sound rather than by consequence. | Rank by consequence, reach and detectability — customer-facing and irreversible actions first. |
You should now be able to
- Distinguish inherent LLM limitations from failure modes introduced by a system’s design.
- Name the specific risks in a scenario — confident errors, injection, stale knowledge, inconsistency, compounding, leakage, bias, over-trust.
- Explain why agents and tool connections add failure modes beyond those of a single model call.
- Run a structured risk review that pairs each risk with a control and an owner.
- Rank risks by consequence, reach and detectability rather than by familiarity.