The model is not magic.

The whole pipeline in one diagram.

Motto: the model takes tokens in, predicts one token out, and does it again until it stops.

Here's everything that happens when you hit send on a Claude prompt:

1. Tokenize. Your text is split into tokens. English averages roughly 1.3 tokens per word (about 4 characters per token) — sometimes a token IS a word, sometimes part of one.
2. Embed. Each token becomes a vector — a list of ~4,096 numbers that captures what it "means" to the model.
3. Attend. Each token's vector is updated by looking at every other token's vector. The output: a refined vector that knows the context.
4. Predict. The model produces a probability over every possible next token. Picks one (with randomness controlled by temperature).
5. Loop. The picked token is appended to the input. Go back to step 1. Stop when the model emits a special end-token or hits your max-tokens cap.

That's it. Everything else — "agents", "memory", "tools", "thinking" — is built on top of this loop. The next nine units unpack each step, so by the end you'll see Claude's behavior as the natural consequence of this pipeline, not a mystery.

Tokens (not words).

Motto: the model thinks in tokens, you write in words; the difference is where most surprises live.

A token is the unit the model reads and produces. It is NOT a word. Roughly:

• Common short English words ("the", "and", "of") are usually one token.
• Longer or less common words split: "unconditionally" → ["un", "conditional", "ly"].
• Numbers, punctuation, whitespace, emoji are their own tokens.
• Programming code is roughly 1.5x denser in tokens than prose — lots of punctuation.
• Non-English languages can be 2-5x denser than English for the same idea.

A trap to avoid: the model cannot reliably tokenize text for you. Ask Claude "split this into tokens" and it will guess — it has no introspective access to its own tokenizer, so its split and its count are often wrong. The only ground truth is the count_tokens API (or the usage field of a real call). That gap is exactly what the next lab measures.

Why this matters

Token count = cost + speed + context fit. You're not paying per character; you're paying per token. Code costs more than prose for the same intent. Long German compound words cost more than the English equivalent. This is the foundation for every cost decision you'll make — so it's worth seeing the real numbers, not the model's guesses.

Copy# count_tokens.py  —  pip install anthropic ; export ANTHROPIC_API_KEY=sk-ant-...
import anthropic

client = anthropic.Anthropic()
sentence = "The quick brown fox jumps over the lazy dog near the riverbank at dawn."
resp = client.messages.count_tokens(
    model="claude-opus-4-8",
    messages=[{"role": "user", "content": sentence}],
)
print("TRUE tokens:", resp.input_tokens)

Embeddings: words become vectors.

Motto: the model doesn't see "cat" — it sees a list of 4,096 numbers that capture "cat-ness."

Every token gets mapped to a fixed-size vector (a list of numbers, usually 4,096 long in modern LLMs). Tokens that mean similar things end up with vectors that point in similar directions. The model never reads characters — it reads these vectors.

What this lets you predict

• Synonyms behave the same. "Big" and "large" have very similar embeddings, so prompts using one work like prompts using the other.
• Typos often survive. "Embedding" and "embeding" have similar embeddings; the model usually understands.
• Wrong-but-related terms confuse predictably. "Java" and "JavaScript" embed in similar regions; the model needs context to know which you mean.

The prompt that demonstrates embeddings

CopyRate how similar these word pairs are in the model's internal
representation, on a 0-10 scale. Explain your reasoning.

1. dog & puppy
2. dog & cat
3. dog & banana
4. teacher & instructor
5. teacher & janitor
6. teacher & equation
7. JavaScript & TypeScript
8. JavaScript & Java
9. JavaScript & jaywalking

Now tell me: which pair would cause the most prompt-rewording
problems in production, and why?

The model can't read out its real embedding distances either (same introspection gap as tokens) — treat its 0–10 scores as predictions to test, not measurements. The point isn't the exact numbers; it's that you can predict which pairs behave alike before you run them.

Attention in one paragraph.

Motto: every token looks at every other token and weighs which ones matter for its own meaning.

"Attention" is the famous trick. Here it is in one sentence: for each token, the model computes a weighted average of all the other tokens' vectors, where the weights are learned to highlight which tokens are relevant.

Concretely: when the model is processing the token "it" in "The dog ran because it was hungry", attention lets the "it" token "look at" the "dog" token strongly (and ignore "ran", "because", etc). The vector for "it" after attention is updated to include "dog-ness." Pronoun resolution = high attention weight from "it" to "dog".

This happens not once but in many layers, with each layer's attention learning different relationships. Early layers pick up syntax; later layers pick up meaning, intent, even style.

What this lets you predict

• Order can matter a lot. Putting instructions at the end vs the start changes which tokens get attended to in the final layer.
• Long context dilutes attention. A 100k-token document drowns the few tokens that actually matter. This is why retrieval beats stuffing.
• Markdown structure helps. Headings give the model strong "section boundary" tokens to anchor attention on.

The next-token-prediction loop.

Motto: the model only ever picks one token at a time; everything else is a loop.

Here is the surprising thing: the model does not "write a response." It picks one token, appends it to the input, and runs the whole pipeline again. A 500-word response is 500+ separate forward passes, each one picking a token based on everything that came before.

Consequences:

• Once a token is picked, it's committed. The model can't go back and revise an earlier word.
• Streaming is real. The reason you see tokens stream out is because they ARE generated one at a time; the API just sends them as they're produced.
• The "think step-by-step" trick works because each thinking token is added to context, which the next prediction can attend to. Writing reasoning out loud literally improves the next answer.
• Stop tokens end the generation. The model emits a special end-of-message token when it decides it's done.

Temperature, top-p, top-k.

Motto: at each step the model has a probability over every possible next token; these three knobs shape how it picks.

After the model computes probabilities for every next-token candidate, it has to pick one. The picking is controlled by:

• Temperature (0 to ~2). Reshapes the probability distribution. 0 = always pick the highest-probability token (deterministic). 1 = use the raw probabilities. Higher = flatter, more random.
• Top-p (0 to 1, default ~0.9). Only consider the smallest set of tokens whose probabilities sum to p. Filters out the unlikely tail.
• Top-k (an integer, often unset). Only consider the top k most likely tokens. Hard cutoff.

Current Claude caveat: these are the classic decoding knobs, but not every current model exposes them. Claude Opus 4.8 (and Opus 4.7) reject non-default temperature, top_p, and top_k — sending one returns a 400 error. For those models you steer with prompting plus adaptive thinking / effort. Models like Claude Sonnet 4.6 and Haiku 4.5 still accept the sampling knobs, so the API half of the lab below targets one of those.

Two ways to feel sampling at work

You can’t set temperature on Opus 4.8 in the chat UI — but the model still samples, so regenerating reveals its built-in variance. The optional API snippet pins temperature explicitly on a model that exposes it.

Copy# sampling.py  —  uses a model that still accepts temperature (Opus 4.8 rejects it)
import anthropic

client = anthropic.Anthropic()
prompt = "Write a one-sentence opening for a short story about a librarian who discovers a hidden room."

for temp in (0.0, 1.0):
    print(f"\n=== temperature={temp} ===")
    for _ in range(3):
        msg = client.messages.create(
            model="claude-sonnet-4-6",   # sampling-capable; or claude-haiku-4-5
            max_tokens=80,
            temperature=temp,
            messages=[{"role": "user", "content": prompt}],
        )
        print("-", msg.content[0].text.strip())

The rule of thumb: where sampling knobs are supported, use low temperature for canonical answers (extraction, classification, code), medium for one good answer with some variation (writing, creative tasks), and high only if you want chaos. Where knobs are not supported (Opus 4.8), ask for the behavior directly in the prompt.

Context window economics.

Motto: context is finite, expensive, and the most-attended tokens are at the start and the end.

Every model has a maximum number of tokens it can hold in context at once (200K, 1M, etc). Three things to know:

1. Bigger windows cost more — linearly on your bill. What you’re billed is tokens × rate, so doubling the input tokens roughly doubles the input cost. (Separately, the attention compute itself grows quadratically — O(n²) in token count — which is why latency climbs faster than price, and why providers cap context. Don’t conflate the two: billed cost is linear, attention compute is quadratic.)
2. Beginning + end get the most attention. Tokens in the middle of a long context tend to be under-weighted. This is empirically observed across most LLMs ("lost in the middle"). Put your important instruction first AND repeat it last.
3. The cache exists. Modern APIs (including Claude) let you cache stable prefixes of your prompt, so re-sending the same system prompt costs ~10% of original. This is the single biggest production cost lever.

A real "lost in the middle" demo — with a real haystack

We ship a ~2,900-word document with one fact planted in the middle. Download it, paste it into Claude, and ask only the question that fact answers. Grade against the key (don’t peek at the key first — it spoils the test).

• haystack.txt — the document (planted fact lives ~58% of the way through). Right-click → Save As, or open and copy all.
• haystack_key.txt — the answer key. Open this after you record your result.

Why prompts cost what they cost.

Motto: cost = input tokens × input rate + output tokens × output rate; output is usually 5x more expensive per token than input.

The cost of a Claude API call is:

total_cost = (input_tokens × input_rate)
           + (output_tokens × output_rate)
           + (cached_input_tokens × cached_rate)  [Claude only]

The output rate is typically ~5x the input rate. This means:

• Long responses are more expensive than long prompts. A 4000-token response costs more than a 4000-token system prompt.
• Cache is your friend. If your system prompt is 4000 tokens and you call the API 100 times with the same prompt + different user inputs, caching saves you ~90% on the system-prompt cost.
• Asking for shorter responses cuts cost AND improves quality. "Answer in ≤3 sentences" is a cost knob, not just a style preference.

The cost-estimation prompt

Copy# Estimate before you build
Estimate the API cost for the following Claude API call. Assume
Claude Sonnet 4.6 pricing (~$3 per 1M input tokens, ~$15 per 1M
output tokens; cached input is 10% of normal input). Check the
current pricing page before using this for a real budget.

System prompt: [paste yours, or use ~2000 tokens of made-up rules]
User message: [paste yours, or describe a typical user request]
Expected response length: [your estimate, e.g. 500 tokens]
Calls per day: [your estimate, e.g. 100]
% of calls reusing the same system prompt: [e.g. 95%]

Give me:
1. Cost per call without caching
2. Cost per call with caching
3. Cost per day at the volume above
4. Where the biggest savings would come from

Copy# cost_check.py  —  read the real usage field after a call
import anthropic

client = anthropic.Anthropic()
msg = client.messages.create(
    model="claude-sonnet-4-6",           # $3 / $15 per 1M tokens
    max_tokens=500,
    messages=[{"role": "user", "content": "<paste your typical request>"}],
)
u = msg.usage
cost = u.input_tokens/1e6 * 3 + u.output_tokens/1e6 * 15
print(f"in={u.input_tokens} out={u.output_tokens}  ACTUAL=${cost:.5f}")

Why models hallucinate.

Motto: the model isn't lying — it's predicting the most likely next token, and "I don't know" is rarely the most likely next token.

Hallucinations have a structural cause. The model is trained on text where humans confidently state things. So when you ask a question, the most likely continuation pattern is "a confident answer," not "an admission of ignorance." Three failure modes:

1. Fact hallucination — the model invents a citation, function name, person, or statistic that sounds right. Cause: the surrounding pattern is "[author/function/year]" and the model fills the slot with a plausible value.
2. API hallucination — the model invents a method on a library that doesn't exist. Cause: the library has many real methods that resemble the made-up one, so the made-up one is locally probable.
3. Reasoning hallucination — the model produces a logically-valid-looking chain that's actually wrong. Cause: each step is locally probable; the global validity isn't checked.

The prevention pattern

CopyWhen answering this question, follow these rules:

1. If you don't know something with high confidence, say so
   explicitly. Use the format: "I'm not sure about X — please
   verify against [authoritative source]."
2. For any factual claim (name, number, date, function name),
   tell me your confidence: HIGH (would bet money), MEDIUM
   (probably right), LOW (could be wrong).
3. For any external reference (paper, function, library, API),
   give me a way to verify it (URL, command to run, doc to check).
4. If I ask for code, tell me which exact library version you're
   targeting and how I can confirm the function/method exists.

Now answer: [your real question]

Why this works: the rules become part of the context the model attends to when picking each next token. Asking for confidence labels forces the model to allocate probability across "high/medium/low" tokens rather than just confident assertion.

What you can debug from outputs alone.

Motto: three observations explain 80% of "why did it do that."

Most "the model is broken" reports turn out to be one of these:

One page on your desk.

The mental model from this practice on one page — with blanks you fill from your own numbers as you go. Print it; keep it next to your monitor for the first month after you start building seriously with Claude. The facts are corrected; the fill-ins make it yours.

Copy it, then fill each __ from the matching unit’s lab as you complete it. By the end you have a one-page model in your own measurements, not a generic copy.

Copy# What Claude is actually doing — my one-page model

## The pipeline
Input text → tokens (~1.3 per word, ~4 chars each) → embeddings (4,096-dim vectors)
→ attention layers (each token reweighted by every other token)
→ probability over next token → pick one (per model sampling rules)
→ append → repeat → stop on end-token or max_tokens
[U01] Pipeline stages I recited cold, Claude marked CORRECT: __ /5
[U02] My test sentence: GUESS __ tokens, TRUE __ tokens, DELTA __
[U03] Java/JavaScript swap changed my answer via: __ (language / libraries / syntax)
[U04] Reply-in-French obeyed — buried: __ / first+last: __ (yes/no)
[U05] Bat-and-ball — terse: $__ / step-by-step: $__ (correct = 0.05)

## The knobs that change behavior
- temperature/top_p/top_k: sampling knobs where supported; Opus 4.8 (and 4.7) REJECT non-default values (400 error)
- adaptive thinking / effort: use these instead of sampling knobs on Opus 4.8
- max_tokens: hard cap on response length
- cache: re-using same system prompt cuts cost ~90%
[U06] Regenerating my creative prompt 3x varied: __ (none / wording / fully)

## The economics
- output tokens cost ~5x input tokens
- BILLED cost grows LINEARLY with tokens (tokens × rate); attention COMPUTE grows quadratically O(n²) — don't conflate them
- shorter responses = cheaper AND usually better quality
- cache stable prefixes; the savings are real
[U08] My typical call ≈ __ in + __ out tokens ≈ $__ /call ≈ $__ /day

## Why hallucinations happen
- the model picks the most LOCALLY probable next token
- "I don't know" is rarely the most probable continuation
- fact / API / reasoning hallucinations have different cures
- fix by forcing confidence labels and verification rules in the prompt
[U09] The fake-package bait gave me: __ (confident fake / honest 'unknown')

## Long-context gotchas
- "lost in the middle": model under-weights tokens in the middle
- put critical instructions first AND last
- prefer retrieval over context-stuffing at scale
[U07] haystack.txt recall — MIDDLE: __ (hit/miss), EARLY: __ (hit/miss)

## When something looks wrong, ask in order:
1. Are sampling settings too loose, or unsupported for this model?
2. Did I exceed max_tokens?
3. Is the important info buried in the middle of context?
4. Did the model invent a fact that sounds right? (check 1-2 with web)
5. Did the system prompt change (cache miss)?
6. Is this a safety refusal? (rephrase intent)
[U10] A real symptom I diagnosed from the table → cause: __

The model is not magic. The model is a token predictor in a loop.
Everything else is engineering on top of that.

This page replaces "the model is acting weird" with a checklist. Past the first week of building seriously, you'll diagnose 80% of issues using the table in Unit 10 + this cheatsheet, before you ever open the docs.