LLM fundamentals

Tokens are the unit everything is measured in

A token is a fragment of text — commonly a short word, part of a longer word, or a piece of punctuation. The model never sees characters or words directly; it reads and writes tokens. That matters to you for three practical reasons: tokens are what you are billed for, tokens are what rate limits count, and tokens are what the context window is measured in. Every response reports the count in its usage field, so you never have to guess after the fact.

Do not estimate token counts from character counts in production code. Text in other languages, code, JSON and base64-encoded images all tokenise at very different rates. The API exposes a token-counting endpoint for measuring a request before you send it — that, and usage on the response, are covered in 5.4.

The context window is a budget, not a memory

The context window is the total text the model can reference while generating a response — including the response. The documentation is explicit about what counts toward it: the system prompt, every message in messages including tool results, images and documents, your tool definitions, and the output Claude generates, extended thinking included. Cached content counts too; caching changes what you pay, not what fits.

There is no memory between requests. A conversation feels continuous only because your code resends the whole history every time, so a long chat costs more per turn than a short one even though the user typed the same amount. Each turn's output becomes part of the next turn's input.

Sampling and non-determinism

Having scored every candidate for the next token, the model has to choose one. Choosing the highest-scoring token every time makes output repetitive; choosing from the distribution makes it varied. The classic controls are temperature — flattening or sharpening the distribution — and the nucleus and top-k cutoffs top_p and top_k, which limit how far down the ranked list a choice can come from. The documented advice for temperature is to sit closer to 0.0 for analytical work and closer to 1.0 for creative work, and to adjust one of temperature or top_p, not both at once.

The critical sentence is in the API reference: results are not fully deterministic even at temperature 0.0. Low temperature narrows variation; it does not remove it. Anything in your system that requires an exact repeat of a previous answer — a test that asserts on an exact string, a cache keyed on output, a workflow that assumes the same category name twice — is built on a promise the model does not make.

Thinking: extended and adaptive

Thinking lets the model produce reasoning tokens before its answer. Those tokens appear as thinking blocks, count toward max_tokens, and are billed as output. There are two modes, and the exam cares about which one a given model takes.

# Extended thinking: you set a fixed budget, in tokens.
resp = client.messages.create(
    model=MODEL, max_tokens=16000,
    thinking={"type": "enabled", "budget_tokens": 10000},  # min 1024, < max_tokens
    messages=messages,
)

# Adaptive thinking: Claude decides how much to think; you set the effort level.
resp = client.messages.create(
    model=MODEL, max_tokens=16000,
    thinking={"type": "adaptive"},
    output_config={"effort": "high"},   # low | medium | high (and higher levels)
    messages=messages,
)

# Either way, reasoning arrives as its own block type.
for block in resp.content:
    if block.type == "thinking":
        pass                                    # internal reasoning
    elif block.type == "text":
        print(block.text)                       # the answer you show the user
print(resp.usage.output_tokens_details.thinking_tokens)

Two behaviours catch people out. Under adaptive thinking, a request may come back with no thinking block at all — that is the mode working, not a failure, and code that expects a thinking block first will break on it. And whether earlier turns' thinking stays in context depends on the model: newer models keep previous thinking blocks, where they are billed as input on later requests, while others strip them automatically. Your token budget therefore depends on the model, not only on your prompt.

Zero-, single- and multi-shot prompting

Since every token is chosen in the light of everything before it, examples in the prompt are an unusually direct lever: they show the model the shape of the answer instead of describing it. Zero-shot gives instructions alone. Single-shot adds one worked example. Multi-shot — few-shot — adds several, and the documentation recommends three to five for best results, wrapped in <example> tags inside an <examples> block so the model can tell them apart from instructions.

Categorise this support email by
urgency and topic.

"My card was declined twice and
I have a flight tomorrow."

Categorise support email by urgency
and topic.
<examples>
<example>
Email: Password reset link expired.
Output: {"urgency":"low",
 "topic":"account"}
</example>
<example>
Email: Charged twice for one order.
Output: {"urgency":"high",
 "topic":"billing"}
</example>
</examples>

Good examples are relevant — they mirror the real inputs — and diverse, so the model does not generalise a pattern you never intended. If every example you supply happens to be a billing complaint, expect billing to be over-predicted. Deliberately include the edge cases you care about; an example is the cheapest way to state a rule you would otherwise have to write three sentences about.

Traps the wrong answers are built from

You should now be able to

• Explain next-token generation and say which application behaviours follow from it.
• Account for everything that consumes the context window, and diagnose a “prompt is too long” failure.
• Describe what temperature, top_p and top_k do, and why output is never fully deterministic.
• Choose between extended and adaptive thinking, and configure budget_tokens or effort correctly.
• Decide when zero-, single- or multi-shot prompting is appropriate and structure examples well.