Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 16 · Theme 3: Dynamics, sequences & world models

Building & probing a language model

Thursday, November 12 · Fall 2026

Reader-friendly version — lecture content with figure descriptions

Where we left off: one attention head

Y=softmax ⁣(QK⊤dk)⏟attention weights AV\mathbf{Y}=\underbrace{\mathrm{softmax}\!\left(\frac{\textcolor{#B5396B}{\mathbf{Q}}\textcolor{#3E6D8E}{\mathbf{K}}^\top}{\sqrt{d_k}}\right)}_{\text{attention weights }\mathbf{A}}\textcolor{#3F7A6B}{\mathbf{V}}

Indices: ii is the token doing the reading (a row of the tables), jj a token being read (a column)

  1. Scores sij=qi⋅kjs_{ij}=\textcolor{#B5396B}{\mathbf{q}_i}\cdot\textcolor{#3E6D8E}{\mathbf{k}_j}: how well token ii's query matches token jj's key, for every pair, divided by dk\sqrt{d_k}
  2. Causal mask on the score table: scores of later tokens (j>ij>i) become −∞-\infty, so their weights are 0
  3. Weights aija_{ij}: the softmax turns each row of scores into positive numbers that add up to 1; a larger score gets a larger weight
  4. Output yi=∑jaij vj\mathbf{y}_i=\sum_j a_{ij}\,\textcolor{#3F7A6B}{\mathbf{v}_j}: token ii's new vector, the values averaged with its weights

From the head's output to the loss

  1. Readout: logits zi=Wyi\mathbf{z}_i=\mathbf{W}\mathbf{y}_i, one per character; probabilities pi=softmax(zi)\mathbf{p}_i=\mathrm{softmax}(\mathbf{z}_i)
  2. Loss at each position ii: −ln⁡-\ln of the probability given to the true next character. The training loss averages these numbers over all positions
  3. Training lowers the loss by changing all the weights: where a head looks is learned

To write text, the trained model draws the next token from pi\mathbf{p}_i, appends it, and repeats (later today)

Model Loss (nats) Perplexity elosse^{\text{loss}}: equally likely characters
Uniform guess 4.57 97
Bigram table 2.28 9.8
Average of earlier characters 2.03 7.6
One attention head 1.23 3.4

Attention in real systems

Today: what trained heads do in real models; then build a whole model from heads, train it to write, and open it up

GPT-2: a language model we can open up

  • GPT-2 (OpenAI, 2019): a transformer, 12 blocks of 12 attention heads, 124 million learned parameters. Its tokens are about 50,000 word pieces (whole words, or parts such as urs), not characters
  • One objective, the same as our one-head model: at every position of about 8 million web pages, predict the next token (a word or part of a word); the loss is −ln⁡p-\ln p of the true next token. No labels, "no tasks"
  • Whatever its heads do, they do because it lowers that loss. ChatGPT starts the same way, then is further trained to follow instructions (how: in two lectures)
  • Inside: embedding → 12 blocks, each attention + a small network → readout. Later today we build a block, then switch heads off (knockout) and copy activity between runs (patch)
  • Its parameters are public: we can run it on a sentence and look at each head's attention weights

A previous-token head

Attention weights of GPT-2 small's block 4, head 11 on the sentence The black cat sat on the mat. The black cat sat on. Rows are the token being updated, columns the token being read; only the lower triangle is filled. A dark stripe runs just below the diagonal: every token after the first puts almost all its weight on the token just before it. Average weight on the token before: 0.999

Real computation: GPT-2 small, weights as trained · head chosen on separate test inputs

  • The input: GPT-2 reads The black cat sat on the mat. The black cat sat on and predicts the next token at every position, as in training. We record one head's weights
  • Each row: one token's attention weights aija_{ij} over itself and earlier tokens; they sum to 1
  • This head: each token after the first reads the token just before it, so its output carries that token's value

An induction head

Attention weights of GPT-2 small's block 5, head 5 on the same sentence. In the first clause almost every row puts its weight on the first token. In the repeated clause the second black reads the first cat and the second cat reads the first sat, the tokens that followed their earlier copies. Average weight on the token after the earlier copy, over the repeated clause: 0.608

Real computation: GPT-2 small, weights as trained · head chosen on separate test inputs

  • Text repeats itself: names, phrases. In a repeat, what followed a token last time is a good guess for what follows it now
  • This head reads exactly that: in the repeat, black reads cat, cat reads sat; in the first half it parks on the first token
  • Why "induction": the text showed A B once; when A returns, the head points to B, a rule induced from the context itself
  • Our one-head model could not do this: it takes two heads in a row (later today)

An attention sink

Attention weights of GPT-2 small's block 9, head 11 on the same sentence. One dark first column: almost every token, in both halves of the sentence, puts almost all its weight on the first token. Average weight on the first token: 0.926

Real computation: GPT-2 small, weights as trained · head chosen on separate test inputs

  • This head: almost every token puts almost all its weight on the first token, in both halves
  • Why: a row's weights must add up to 1, so a head with nothing useful to read still puts them somewhere
  • What it does: its output is then the first token's value, nearly the same everywhere: the head contributes almost nothing new
  • Vision transformers do the same, on background patches; spare "register" tokens give the weight a place to go (Going further)

Try it: all 144 heads of GPT-2

From a head to a model

Attention moves information between positions. What else does a language model need?

The whole model, from the last lecture

Bottom to top, one column per token:

  1. Embedding + position: token → vector
  2. Attention (gold): mixes positions
  3. MLP: each position alone
  4. 2 and 3 repeat in LL blocks
  5. Readout: a probability for every possible next token

Next: several heads, the MLP, the wiring

Schematic of a transformer, read from the bottom up, with one column per token. At the bottom, six tokens, the characters t, h, e, a space drawn as an underscore, c and a. Each becomes a small blue vector, labelled embedding plus position. A grey line runs up from each vector, the token's residual stream. Inside a dashed frame marked times L blocks, a gold self-attention band spans all six columns: lines from every column fan in to the last one, so the last token reads itself and the five earlier tokens and none after it. Above it, each column has its own small box labelled MLP, the small network that works on each token alone. At the top, an output layer and softmax spans the columns and gives each column's predicted next token: h, e, space, c, a, t, the input shifted by one

Multi-head attention

Several heads let one block read in several ways at once. Run nhn_{\mathrm{h}} heads side by side:

Y(h)=Attention ⁣(XWQ(h), XWK(h), XWV(h))\mathbf{Y}^{(h)}=\mathrm{Attention}\!\big(\mathbf{X}\mathbf{W}_Q^{(h)},\,\mathbf{X}\mathbf{W}_K^{(h)},\,\mathbf{X}\mathbf{W}_V^{(h)}\big)

MHA(X)=Concat(Y(1),…,Y(nh)) WO\mathrm{MHA}(\mathbf{X})=\mathrm{Concat}\big(\mathbf{Y}^{(1)},\dots,\mathbf{Y}^{(n_{\mathrm{h}})}\big)\,\mathbf{W}_O

  • Recall: dmodeld_{\text{model}} is the width of every token's vector. The one-head model had dmodel=128d_{\text{model}}=128, and its queries, keys and values were 128 wide too
  • Now the width is split among the heads: dmodel=nh×dkd_{\text{model}}=n_{\mathrm{h}}\times d_k. The model you train: dmodel=256d_{\text{model}}=256, nh=4n_{\mathrm{h}}=4, so each head's queries, keys and values have dk=dv=64d_k=d_v=64 numbers
  • A query and a key always have the same length, dkd_k, because a score is their dot product; values have length dvd_v
  • WO\mathbf{W}_O (dmodel×dmodeld_{\text{model}}\times d_{\text{model}}) mixes the concatenated outputs back to width dmodeld_{\text{model}}

Shapes at a glance

Quantity Shape In your model One row is
X\mathbf{X} T×dmodelT\times d_{\text{model}} T×256T\times 256 one position's vector
Q, K\textcolor{#B5396B}{\mathbf{Q}},\ \textcolor{#3E6D8E}{\mathbf{K}} (one head) T×dkT\times d_k T×64T\times 64 one query, one key
V\textcolor{#3F7A6B}{\mathbf{V}} (one head) T×dvT\times d_v T×64T\times 64 one value
scores, weights T×TT\times T T×TT\times T one query against every key
one head's output T×dvT\times d_v T×64T\times 64 an average of values
MHA(X)\mathrm{MHA}(\mathbf{X}) T×dmodelT\times d_{\text{model}} T×256T\times 256 same shape as X\mathbf{X}
  • TT: tokens in the window (up to 256) · dmodel=256d_{\text{model}}=256 · 4 heads, so dk=dv=256/4=64d_k=d_v=256/4=64
  • In code, BB windows are processed together, a batch: every shape gets BB in front, (B,T,256)(B,T,256)

A small network for each position

Attention mixes information across positions. A two-layer network then works on each position on its own, with the same weights everywhere:

MLP(x⟨t⟩)=W[2] GELU ⁣(W[1]x⟨t⟩+b[1])+b[2]\mathrm{MLP}\big(\mathbf{x}^{\langle t\rangle}\big) = \mathbf{W}^{[2]}\,\mathrm{GELU}\!\big(\mathbf{W}^{[1]}\mathbf{x}^{\langle t\rangle} + \mathbf{b}^{[1]}\big) + \mathbf{b}^{[2]}

Left: the GELU transfer function, Gaussian Error Linear Unit, plotted for inputs from minus 4 to 3, with the ReLU dashed for comparison. GELU is close to 0 for negative inputs, dips slightly below 0 around minus 1, and becomes nearly equal to its input for positive inputs. Right: the widths of the small network in the assignment's model, drawn to scale: a bar for the input x of 256 numbers, an arrow W[1], which maps 256 numbers to 1,024, to a bar four times longer, the hidden layer of 1,024 numbers with GELU applied to each, and an arrow W[2], which maps them back, to a 256-number output

  • GELU, Gaussian Error Linear Unit: a smooth ReLU, GELU(z)=z Φ(z)\mathrm{GELU}(z)=z\,\Phi(z)
  • The hidden layer is 4 × wider than dmodeld_{\text{model}}; it holds most of the model's weights

Residual connections and layer normalization

  • Residual: each sublayer adds its output to its input, X←X+Sublayer(X)\mathbf{X}\leftarrow\mathbf{X}+\mathrm{Sublayer}(\mathbf{X}), as in ResNets. The stream runs straight through; attention and the MLP only add to it
  • LayerNorm: rescale each position's vector to mean 0 and variance 1, then a learned scale and shift
  • Pre-norm: LayerNorm at the start of each branch, not on the stream
  • One block = both branches. Blocks stack: the model you train has 4

One pre-norm transformer block, read from the bottom up. The residual stream x enters at the bottom and runs straight up the left side to the next block. A first branch leaves the stream, goes through LayerNorm and then multi-head attention, and its output is added back to the stream at a plus sign. A second branch then leaves the stream, goes through LayerNorm and the small network (MLP), and is added back at a second plus sign. LayerNorm sits at the start of each branch, never on the stream itself

Why copying takes two heads in a row

Two bands of word boxes for The black cat sat on the mat. The black, the last black marked as the current position. Top band, block 1, a previous-token head: an arrow from each word to the next, and the first cat receives a tag reading previous: black. Bottom band, block 2, an induction head: the query of the last black, find previous = black, matches the key of the first cat, previous: black, along a curved gold arrow; the value of cat, cat, flows to a box reading predict: cat. A faint dashed arrow from the last black back to the first black is labelled one head alone: finds black, reads back black

  1. Problem (dashed): one head finds the earlier black, but reads back black
  2. Block 1: each token notes the token before it; the first cat gets "previous: black"
  3. Block 2: the second black seeks "previous: black", finds cat: predict cat

Try it: inside a transformer

Training a model to write

What do we train it to do, and how do we know it learned anything?

One objective: predict the next character

  • Text gives its own labels: at every position tt, the target is the character y⟨t+1⟩y^{\langle t+1\rangle} that comes next
  • The last layer outputs, at each position, a probability for every character: y^⟨t⟩\hat{\mathbf{y}}^{\langle t\rangle}
  • Loss: cross-entropy, averaged over positions, in nats

L=−1T∑t=1Tlog⁡y^ y⟨t+1⟩⟨t⟩\mathcal{L}=-\frac{1}{T}\sum_{t=1}^{T}\log \hat{y}^{\langle t\rangle}_{\,y^{\langle t+1\rangle}}

  • With the causal mask, one forward pass gives all TT predictions: TT training examples at once

Two rows of character boxes. The bottom row, in blue, is the input: t, h, e, underscore for a space, c, a, t, underscore, s, a. The top row, in green, is the target at each position: the next character, h, e, underscore, c, a, t, underscore, s, a, t. A gray arrow runs up from each input box to its target

Batches and the training loop

  • Cut random windows of T+1T+1 characters from the training text; input = the first TT, target = the last TT
  • Stack BB windows: inputs of shape (B,T)(B,T), logits of shape (B,T,K)(B,T,K)
  • One step: clear the old gradients, run the model forward to the logits, compute the cross-entropy over all B×TB\times T positions, call backward(), take one step of AdamW (Adam with weight decay). The cross-entropy applies the softmax itself
  • Every few hundred steps, measure the loss on held-out text
  • Landmarks: a uniform guess scores ln⁡K\ln K (ln⁡97=4.57\ln 97=4.57 nats for 97 characters); a bigram table scores far lower, a trained transformer lower still
  • Recall the perplexity elosse^{\text{loss}}: the number of equally likely characters that would give the same average loss (e4.57=97e^{4.57}=97 for the uniform guess)

Generating text: sample, append, repeat

  1. Feed in a prompt
  2. Read the model's probabilities for the next token, at the last position
  3. Draw one token from them and append it to the prompt
  4. Repeat; keep only the last TT tokens if the text grows longer than the context
  • Temperature: divide the logits by a number before the softmax. Below 1: the likely tokens get likelier, predictable text; above 1: the odds even out, more varied text, then nonsense
  • Temperature near 0 is greedy (always the most likely token): it tends to loop
  • Every new token takes one more pass through the whole model: a 1,000-token reply is 1,000 passes

Temperature in a real model

Prompt: The black cat sat on the mat. Then · GPT-2 small, 40 new word pieces, same seed

  • Greedy (always the most likely piece): he looked at the white cat. "You're not going to be able to see me," he said. "I'm not going to be able to see you," the white …
  • Temperature 0.7 (≈ 7 choices for the first piece): a bright flash of light shot through the night sky. A small man was squatting on the ground next to him. He had a long blond hair. His eyes were a large hematite.
  • 1.0 (≈ 42): a button seven came down and… A black cat the size of the Olympic flame moved out of the end of the arm she so enjoyed to sit on.
  • 1.5 (≈ 2,630): along came seven more faces perched… Goldbacks Skooter bipartisan candidates spill radical words for CathP wrestler Sly Blastin …

Real computation: GPT-2 small, pure sampling, torch seed 0; "≈ 7": like a choice among 7 equally likely pieces

  • Same model, same probabilities: how you draw decides the text. Too low loops, too high is nonsense; chat systems use 0.7–1 and top-p (keep the likeliest pieces up to probability p, rescale, draw)

What did the model learn?

Read its heads, then lesion them

Three ways to open a model

  • Read: look at where each head puts its weights; or fit a readout on the residual stream after each block, as you did on neural populations. If it works on held-out text, that block holds the information, though the model may not use it
  • Lesion: switch a part off and measure what fails, compared with a control
  • Patch: copy a part's activity from one run into another and measure what comes back
  • Reading suggests a hypothesis; lesions and patches test it

What heads do, and what their weights cannot tell you

  • In trained language models, single heads track relations people can name: verb to its object, determiner to its noun, pronoun to its referent
  • An attention map shows where a head reads. It does not show why the output is what it is
  • As with attribution maps in the explainability lecture, a map is a hypothesis; changing the input or knocking the head out tests it

A behavior to explain: copying a new name

  • A child named Zefam. The sun was warm. Later, Zefam smiled. No training story contains "Zefam"; the second time, a model can predict the letters e, f, a, m well only by copying the first mention
  • Control: A child named Molly. The sun was warm. Later, Zefam smiled. The second name is still Zefam, but no Zefam came before it: nothing to copy, so e, f, a, m must be guessed
  • Benefit (nats) = loss on efam in the Molly version − loss on efam in the Zefam version: large if the model copies, near 0 if not
  • The one-head model (one attention layer): benefit about 0; it cannot copy ("Why copying takes two heads in a row"). The model you train, 4 blocks of 4 heads, can form a previous-token + induction pair: large benefit. Which heads do it?

Knock out a head: a lesion study

  • Knock out one head: replace its output with zeros, so it adds nothing to the residual stream, and run the model again; the other heads are untouched
  • Measure the loss on the behavior the head is thought to support, and on ordinary text as a control
  • A claim that a head is specific needs three results: a large effect on the behavior, a small one on ordinary text, and a small effect when other heads are knocked out instead
  • If knocking out a head changes nothing, another head may be doing the same job. Switch the head back on before the next test

The same lesion in GPT-2

Two bar charts of GPT-2 small's loss in nats per token. Left, copying the second half of a repeated random sequence: intact 0.25, the 8 heads with the highest induction weight knocked out 7.49, 8 random other heads knocked out 0.81 with a range bar up to 2.7. Right, ordinary English sentences: intact 4.69, induction heads out 5.25, random heads out 4.83

Real computation: GPT-2 small, 144 heads, ranked by induction weight on repeated random tokens; knockouts zero the head's output

  • Knock out the 8 heads with the highest induction weight: loss on a repeated random sequence rises from 0.25 to 7.49 nats (a uniform guess is 10.8); loss on ordinary text rises by 0.56 nats. Knock out 8 random heads instead: 0.81 and 4.83 nats. Large where predicted, small elsewhere, with a control

The converse test: put the activity back

Three rows of character boxes for a short story, each ending in the second mention Z e f a m. Row 1, Clean (donor): the earlier name is Zefam; loss on efam low. Row 2, Corrupted: the earlier name is Molly; loss on efam high. Row 3, Patched: the corrupted story, with gold arrows bringing one head's output at the four positions Z, e, f, a of the second mention down from the clean row; loss on efam: question mark. Below: earlier letters are the true text, the model is not writing the name freely

  • A knockout shows whether a part is needed. A patch shows whether the information is in a part's activity: copy that activity from the clean run (first mention Zefam, the donor) into the corrupted run (first mention Molly)
  • Recovery =(Lcorrupt−Lpatch)/(Lcorrupt−Lclean)=(\mathcal{L}_{\text{corrupt}}-\mathcal{L}_{\text{patch}})/(\mathcal{L}_{\text{corrupt}}-\mathcal{L}_{\text{clean}}): 0 if nothing comes back, 1 if all of it does

What attention gains, and what it costs

  • Long range, content-dependent: any position reaches any earlier position in one step, with weights computed from the input itself, so the links change with every input
  • Fast to train: the loss at every position of a window is computed at once, in a few matrix products; the causal mask makes this possible
  • Scale: GPT-2 small reads 1,024 tokens, about two pages. Current models read up to a million tokens, about 750,000 words or eight novels, with the mechanism unchanged
  • The price: a window of TT tokens needs T×TT\times T scores, so doubling the length quadruples the attention computation, and the keys and values of every earlier token stay in memory. A long window does not mean the model uses all of it: accuracy is lowest when the relevant text sits in the middle

Recap

  • Trained heads take on roles you can name: read the previous token, copy what followed the current token last time (induction), or park the weight on the first token (sink).
  • A block: LayerNorm, multi-head attention, add; LayerNorm, a small network per position, add. Blocks stack along the residual stream; the readout gives the next-token probabilities
  • Training: random windows, the next-token cross-entropy at every position at once, checked on held-out text. Writing: sample, append, repeat, with a temperature
  • Opening a model: read the weights or a readout to form a hypothesis, then knock out and patch back to test it, with controls. A circuit found in a small model is a hypothesis for a large one

Minute paper

Submit on Canvas → Minute papers → Minute Paper 17 (access code read out in class).

Write three brief points in your own words:

  1. A language model is only trained to predict the next word. How can it end up
    copying a name it has seen once earlier in a story?
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Further reading for the research trail

Optional: a few research-trail entries over the semester. Any paper mentioned in any lecture qualifies, but papers from these lectures are preferred.

Going further

Optional: heads in long documents and in vision, sampling in real models, another circuit

Twenty heads find a fact in a long document

Figure 1 of Wu et al.: two grids of needle-in-a-haystack accuracy, context length from 5K to 100K tokens across, depth of the hidden sentence from 0.1 to 0.9 down. Left, with the top 20 retrieval heads masked, many cells turn yellow and red, accuracy 63.6. Right, with 20 random heads masked, almost every cell stays green, accuracy 94.7

  • LLaMA-2 (7 B) must recall a fact hidden in 5,000–100,000 tokens. Without its top 20 retrieval heads, accuracy drops to 64%; without 20 random heads, 95%
  • Retrieval head: while answering, its strongest weight is on the word being copied

The same mechanism in vision

Figure 1 of Caron et al.: eight photographs, each followed by the self-attention map of a DINO vision transformer, bright where the class token puts its weight; the bright patches outline the bird, the toothbrush, the building, two giraffes, a boat, a bicycle, a dog and a plane, with no label, mask or box given

  • DINO, a vision transformer trained without labels: each 8 × 8-pixel patch is a token, and the heads work as they do on words. The attention of one extra summary token (the class token) outlines the object
  • Small patches give fine maps: a 480-pixel image is 60 × 60 = 3,600 patches, one weight each

Where the bright patches come from

  • In large vision transformers (DINOv2, OpenCLIP), a few background patches get very large vectors, and other heads put most of their weight on them: attention sinks, like GPT-2's first token
  • The cause is a handful of neurons in one block's small network. Each outputs one number per patch, and at those few patches the numbers are huge
  • Moving them: the image is untouched. While the model runs, edit the tokens: in each patch's vector, set those neurons' outputs to 0 where they were huge, and to the huge value at patches you choose: a patch, as in the assignment
  • Or put the huge values in one extra token added to the input, a register

Attention sinks in vision, and what they hide

Figure 1 of Jiang et al.: an eagle and a dog, then attention maps from OpenCLIP and DINOv2 in three versions each. Original: a few bright background patches and little else. Shifted: the bright patches rearranged into a smiley face or a heart. With a test-time register: the map outlines the bird and the dog

  • Original: sinks on the background · Shifted: inside the network, the authors wrote the huge values into the tokens of patches forming a smiley or a heart, and the sinks appear exactly there · With a test-time register: written into the extra token, and the map shows the animal

The same recipe in real systems

Does the toy recipe survive scale?

Try it: temperature and sampling

The loss ladder with real models

Horizontal bars of held-out loss in nats per character on the same 40 TinyStories stories: uniform guess 4.57, bigram table 2.30, the assignment's 4-block character GPT 0.60, GPT-2 small 0.60, GPT-2 medium 0.53, GPT-2 large 0.49, and TinyStories-33M, trained on the TinyStories training split, 0.34

Real computation: the same 36,000 held-out characters scored by every model; word-piece losses converted to nats per character

  • The loss table of slide 3, continued: a 3-million-weight character model like the assignment's ties GPT-2 small on children's stories; GPT-2 large, 230 × larger, is only 0.11 nats per character lower; a 33-million-weight model trained on such stories beats them all

A different behavior, a different circuit

Figure 1a of Yin and Steinhardt: few-shot in-context-learning accuracy of Pythia 6.9B as a growing percentage of its attention heads is ablated. Clean and random-ablation lines stay flat near 0.56; ablating induction heads lowers accuracy slowly to 0.44 at 20 percent; ablating function-vector heads drops it to about 0.32 at 3 percent, 0.20 at 9 percent and 0.18 at 15 percent

In the figure: ICL = in-context learning, a task learned from examples in the prompt · FV = function-vector heads · ablation = knockout

  • Copying needs induction heads. Learning a task from a few examples is different: in 12 models of up to 6.9 billion weights, knocking out induction heads lowers its accuracy a little; knocking out function-vector heads, which encode which task the examples show, cuts it to about a third
  • A circuit found in a small model is a hypothesis for a large one

- One head turns each token's vector into a new vector that mixes in information from earlier tokens. The formula does it for all tokens of the window at once; the four steps below it read the formula from the inside out - Before step 1, each token's vector x_i is turned into three vectors by three learned matrices: a query q_i = W_Q x_i (what this token looks for), a key k_i = W_K x_i (what this token offers to be found by), and a value v_i = W_V x_i (what this token passes on when it is read). Stacking them for all tokens gives Q, K and V - Step 1, scores. s_ij = q_i · k_j is a dot product: large when token i's query and token j's key point the same way. Dividing by √d_k (d_k is the length of a query) keeps the scores from growing with the vector length. Computed for every pair at once, the scores form S = QKᵀ, a T × T table with one row per reading token - Step 2, causal mask. Before the softmax, every score for a later token (j > i) is set to minus infinity. The mask is applied to the score table S, after the product QKᵀ. Its effect: when the model predicts what comes after token i, it cannot look at the answer - Step 3, weights. The softmax takes one row of scores, exponentiates each one and divides by the row's total: a_ij = e^{s_ij/√d_k} / Σ_{j'≤i} e^{s_ij'/√d_k}. The results are positive, add up to 1, and keep the order of the scores; a masked score becomes a weight of exactly 0. The table of weights is A, and A is what every head figure today shows - Step 4, output. y_i = Σ_j a_ij v_j: token i's new vector is the average of the values, weighted by its row of A. If a row puts all its weight on one token, the output is that token's value: the head has read that token. Y = AV does this for every token at once - Indices: i always names the token doing the reading, j a token being read. In the tables S and A, row i belongs to token i and column j to token j; so a_ij is how much token i reads token j, and the mask keeps only j ≤ i - Same notation as the lecture on attention: S scores, A weights, Y outputs

- This slide closes the loop from the last lecture: how the head's output becomes a prediction, and how the prediction is scored - Readout, step by step: the output y_i is multiplied by a learned matrix W with one row per character of the vocabulary (97 rows). Each row gives one number, the logit z for that character: how strongly y_i points toward it. The softmax then exponentiates the 97 logits and divides by their total, so they become probabilities that add up to 1. The result p_i is the model's guess for the character at position i + 1. This is the same softmax as in step 3 of the previous slide, applied to logits instead of scores - Loss at position i: minus the natural log of the probability given to the character that actually comes next. A sure, correct guess (p = 1) costs 0 nats; a uniform guess over 97 characters (p = 1/97) costs ln 97 = 4.57 nats. The loss of a window averages this over all its positions; this average is the cross-entropy - Training: the gradient of the loss flows back through the readout, through the weighted average, into W_Q, W_K and W_V. So the scores, and therefore the weights a_ij, change during training. Nobody tells a head where to look; it ends up looking wherever lowers the loss - The table: held-out loss on the same TinyStories text for four predictors, from the previous lecture's slide "Back to the bigram table" - Why take the exponential of the loss: the loss is in nats, a log scale that is hard to picture. e^loss undoes the log and turns it into a count. If a model spreads its probability evenly over k characters, the true one gets p = 1/k and the loss is −ln(1/k) = ln k; so e^loss = k. For any model, e^loss is the number of equally likely characters that would give the same average loss: the model is "as uncertain as" a fair choice among that many. The uniform guess over 97 characters has loss ln 97 = 4.57 and e^4.57 = 97; the bigram table, 2.28 nats, is like choosing among 9.8 characters; one attention head, 1.23 nats, among e^1.23 ≈ 3.4 - The same quantity has a name in the literature, perplexity. Because the loss is an average of −ln p over positions, e^loss is also one over the geometric mean of the probabilities given to the true next characters: a perplexity of 3.4 means the model gave the right character, on average in this geometric sense, a probability of about 1/3.4 ≈ 0.29 - It measures the model's average surprise at the real text, not how many characters it actually considers: a model that is confidently wrong can have a large e^loss - Read the last two rows: averaging the earlier characters gives every one the same weight (2.03 nats); one attention head chooses which earlier characters to read, with weights computed from the text itself, and reaches 1.23 nats. Using more of the context, selectively, lowers the loss - The model you build in the assignment stacks such heads into blocks, trains them on the same loss, then knocks heads out and patches activity back, as the rest of today does

- GPT-2 (Radford et al., 2019) is the same kind of model as ours, scaled up: tokens are word pieces (a vocabulary of 50,257) instead of characters, 12 blocks of 12 heads instead of one head, 124 million parameters in the small version - Its training is exactly the objective of the slide "From the head's output to the loss": at every position of WebText (text from about 8 million web pages linked from Reddit, about 40 GB), predict the next token, and lower −ln p of the true next token. Nobody labelled anything, and no head was assigned a job - This is the key to reading the next slides. Whatever pattern a head shows, it was not asked for; training kept it because it made the next token easier to predict. When we say a head "copies", we describe what it does, not a task it was given - ChatGPT and similar chatbots start in the same way, on far more text, then are fine-tuned on examples of instructions and answers and on human ratings of their replies - GPT-2 small is public and small enough to run on a laptop, which is why so much work on how transformers compute uses it. On the next three slides we give it one sentence, let it predict as usual, and record the weight table A of three of its 144 heads

- What is done: GPT-2 reads "The black cat sat on the mat. The black cat sat on" and, as always, predicts the next token at every position. We only record one head's weights A on the way. The sentence was chosen because its second half repeats its first, which the next slide needs - How to read the matrix: one row per token doing the reading (its query), one column per token being read (its key). Color is the attention weight a_ij of slide 2, the softmax output, from 0 (white) to 1 (dark); never the raw score s_ij. Every "weight" on these three slides means a_ij. Each row adds up to 1. The upper triangle is empty because of the causal mask. A dot · marks a token that begins with a space - This head: a dark stripe just below the diagonal. Every token puts almost all of its weight on the token right before it, so its output y_i is almost exactly the value of token i − 1. The first token, The, has nothing before it and reads itself - The number under the figure: the weight each token puts on the token before it, averaged over the rows of this sentence: 0.999 - How this head was found: not by hand. On 20 ordinary English sentences, separate from this one, we measured that same average for each of the 144 heads and kept the largest (0.999). Block 4, head 11 means the fifth block and the twelfth head, counting from 0 as code does - Why a model would learn this: on its own it predicts little, but it writes "the token before me was X" into every position, which a head in a later block can use. The slide "Why copying takes two heads in a row" shows how

- The same sentence, the same kind of matrix, a head in the next block - The motivation: text often repeats itself, a name, a phrase, a line of code. In a repeat, the best guess for the next token is whatever followed the same token last time. A model trained only to predict the next token gains by reading that position, and training found heads that do. "Copying" is the usual name for this behavior - Reading the matrix, first half: no token has appeared before, so there is nothing to find; almost every row puts its weight on the first token, the default place to park weight (next slide) - Second half: the second black puts its weight on the first cat, the second cat on the first sat, and sat and on, more weakly, on on and the. Each token reads the token that followed its own earlier occurrence. The head's output at the second black therefore carries information about cat, which is exactly the token that comes next. How strongly that output pushes the prediction toward cat depends on the head's value and output weights, which the matrix does not show - Why "induction": the head completes a pattern from a single example in the text. The text showed A B once; when A comes back, the head makes B more likely. The name, from Olsson et al. (2022), echoes inductive inference: a rule inferred from the context itself - The number under the figure: the weight each token of the repeated clause puts on the token that followed its earlier copy, averaged: 0.608 - How this head was found: GPT-2 read 50 random tokens followed by the same 50 again, ten times with different random tokens. For each of the 144 heads we averaged the weight each token of the second copy puts on the token after its earlier copy, and kept the largest (0.93). Random tokens ensure the head uses only the repeat, not meaning or grammar - What this gives a model: it can continue anything the text has already said once, including names it never saw in training. Such heads appear early in training in every model Olsson et al. studied - The last bullet sets up a question for later today: our one-head model could not do this. Finding "the token after the earlier black" needs to know, at the earlier cat, that black came before it, which is what the previous-token head writes. So it takes two heads in a row

- The same sentence, a third head. One dark column: almost every token, in both halves, puts almost all of its weight on the first token. The number under the figure is that weight, averaged over the rows: 0.926 - What the sink does: when a row puts its weight on the first token, the head's output there is the first token's value vector, which is nearly the same at every position and which training keeps small. So the head adds almost nothing new: it is a way for a head to do nothing at a position where it has nothing to contribute. A head can be a sink on some text and do real work on other text, as the induction head shows - Is it like the registers? Yes, it is the same phenomenon. In large vision transformers, heads with nothing to read park their weight on a few uninformative background patches; Darcet et al. (2024) added spare tokens, "registers", that take that role, and the maps of the other patches became clean. In a language model the first token plays the role of a register. The optional slides "Where the bright patches come from" and "Attention sinks in vision, and what they hide", in Going further, show it - Why a head would do this: the softmax makes every row add up to 1, so a head cannot put zero weight everywhere. When a head has nothing useful to read at a position, its weight must still go somewhere. On the account of Xiao et al. (2024), trained models learn to send it to the first token, whose key becomes a parking place and whose value adds little to the output. The head contributes little at that position, without its output being set to zero - Compare the induction head: it parked on the first token only in the first half, where there was nothing to copy, and read the repeat in the second half. This head parks everywhere on this sentence - How this head was found: on the same 20 sentences, the weight each token puts on the first token, averaged, for every head; this head has the largest (0.983) among the heads that do not copy (induction weight below 0.1). Some heads do both: the head with the very largest first-token weight also copies when the text repeats, and would look like the induction head here - Why it matters in practice: keeping the first few tokens in memory, together with a window of recent ones, lets a model keep generating stably on an endless stream of text. The same thing happens in vision transformers (see the optional "Going further" slides)

- What the demo is: GPT-2 small's attention weights, computed once, for all 144 heads on four short sentences, plus three average weights and a knockout result for every head. Nothing is computed live; every number comes from the same model as slides 6 to 8 - Left panel: the attention weights a_ij of the selected head on the chosen sentence, the same quantity as slides 6 to 8 (the softmax output, never the raw scores). Row i is the token reading, column j the token read, and each row sums to 1; a dot marks the token each row reads most - Middle panel: this is not an attention matrix. It is a map of the whole model, with one cell per head: 12 rows for the blocks (0 to 11) and 12 columns for the heads in each block (0 to 11), so 12 × 12 = 144 heads. Numbering starts at 0, as in code, which is why the labels stop at 11. The color of a cell is one number about that head, chosen with the buttons above (its induction weight, previous-token weight, first-token weight, or copying damage when knocked out); a star marks the strongest head. Clicking a cell selects that head and redraws the left panel - The "strongest" buttons under the block and head menus jump to the head with the largest previous-token, induction or first-token weight. Right panel: the selected head's three average weights, each with its rank among the 144, and what knocking it out costs (the losses, the intact values, and the increase as a bar) - Walkthrough 1, the previous-token head. Under the block and head menus, press "previous-token" (next to "strongest"): block 4, head 11, the stripe of slide 6; previous-token weight 0.999, rank 1/144 - Walkthrough 2, the induction head on a name. Press "induction" (block 5, head 5), then the sentence "A name repeated": Mr Dursley went home. Later, Mr Dursley. GPT-2 cuts Dursley into three word pieces, D, urs, ley. In the second mention, D reads urs, urs reads ley, and ley reads went: at each step the head points to what followed last time, and at the end to went, the word that followed the name before - Walkthrough 3, induction is spread out. Color the grid by induction: several heads in blocks 5 to 7 light up (block 7, head 10 at 0.92 and block 6, head 9 at 0.91, close to block 5, head 5 at 0.93). The model has several heads that do the same job - The color scale under the grid: each cell's color is one number about that head, from light (0) to dark blue, and the label says which number. For "induction", "previous-token" and "sink" it is an average attention weight, so it runs from 0 to 1: 0 means the head never reads that position, 1 means it always puts all of its weight there. For "copy knockout" it is the increase in loss when the head is removed, from 0 to the largest increase of any head (0.47 nats) - What the knockout panel is: a lesion experiment on one head, the method of the slides "Knock out a head: a lesion study" and "The same lesion in GPT-2" later today, and of the assignment. Knocking out a head means setting its output to zero, so nothing it computes reaches the rest of the model; every other weight stays as trained. Then the model's loss is measured again on two tests, and compared with the intact model - Concretely: the head's output vectors (its weighted averages of values, one per token) are replaced by zeros before the twelve heads' outputs are combined, so that head adds nothing to the residual stream; the other heads and the rest of the model run as usual on what remains - Test 1, copying: 50 random tokens followed by the same 50 again; the loss on the second half. The intact model predicts it well (0.25 nats per token), because induction heads read what followed last time. Test 2, ordinary text: 20 English sentences (intact 4.69 nats per token). Two tests tell whether removing the head hurts copying in particular or everything - Reading the panel: "Copying loss 0.41 (intact 0.25)" means the loss rose by 0.16 nats without this head; the bar draws that increase, and the rank says how many of the 144 heads hurt copying more. The second line does the same for ordinary text. Both bars use one scale so they can be compared - Walkthrough 4, knock a head out. For block 5, head 5: copying rises by 0.16 nats, ordinary text by about 0.05. For the median head, copying rises by only 0.008: because several heads do the job, knocking out one of them costs little. The slide "The same lesion in GPT-2" knocks out eight together - Walkthrough 5, the knockout map. Color the grid by "copy knockout": each cell now shows how much the copying loss rises when that one head is removed. The darkest cell, the head whose removal hurts copying most, is block 0, head 1: copying loss 0.73 instead of 0.25. It is in the very first block, far below the induction heads of blocks 5 to 7 - Why a head in block 0 matters for copying. Select block 0, head 1 on the repeated phrase. In the first half each token reads itself. In the repeat, each token splits its weight about evenly between itself and its own earlier copy: the second black puts 0.54 on itself and 0.45 on the first black. So this head writes into each position a mark that "this token has occurred before". Later heads, including the induction heads, read the residual stream that contains that mark. Remove it, and the copying circuit higher up loses one of its inputs. Wang et al. (2023) call such a head a duplicate-token head - The lesson: on its own, this head's matrix (mostly the diagonal) does not look like copying at all, and its three average weights are near zero. Only the knockout shows that copying depends on it. Which parts a behavior depends on is found by intervening, not by looking at attention maps - Walkthrough 6, heads mix roles. Press "sink": block 5, head 1, the largest first-token weight (0.998). On the repeated phrase it also reads cat and sat, like an induction head; this is why slide 8 shows a different sink - Walkthrough 7, when there is nothing to copy. Choose "A repeated frame": Paris is the capital of France. Rome is the capital of. At the final of, block 5, head 5 does not read France, the token that followed of last time; it parks its weight on the first token. Copying France would give the wrong answer here - At home: the "Two names" sentence. At the final to, the right answer is Mary, the name not repeated. A few late heads put their weight on Mary there: block 9, heads 6 and 9, block 10, head 7, and block 11, head 10 (0.36 to 0.58). Wang et al. (2023) traced this behavior in GPT-2 small and called block 9, head 6 and block 9, head 9 "name movers"

- Last lecture built one attention head: queries, keys and values, the softmax weights, the causal mask and positional embeddings. Today we assemble whole blocks, stack them, train the model to write, and then take it apart to see what its heads do - This is the model you build and train in the fourth assignment

- This is the figure from the end of the last lecture. Today we fill in its parts; each of the next slides zooms into one of them, and the slide "Residual connections and layer normalization" shows how they are wired into one block - Bottom: each token (here a character) is looked up in the embedding table and given a position embedding; the result is a vector with d_model numbers. The grey line above each token is its residual stream: the vector that every later step reads from and adds to - Gold band: attention. It is the only place where a token's vector takes in information from other tokens, here the last token reading itself and the five before it. The next slide replaces the single head of the last lecture with several heads side by side - Boxes: a small network (MLP) applied to each column separately, with the same weights in every column (slide "A small network for each position") - The dashed frame, × L: attention plus small network form one block, and blocks are stacked; the model you train has L = 4, GPT-2 small 12 - Top: a readout (a linear layer and a softmax) turns each column's final vector into probabilities for the next token. Each column predicts the token one step to its right: h after t, e after h, and so on - The readout in detail: after the last block, a final LayerNorm, then a linear layer that maps the d_model numbers to K logits, where K is the vocabulary size (97 characters in our model, 50,257 word pieces in GPT-2), gives one logit per token of the vocabulary; a softmax turns the K logits into probabilities. The same readout is applied at every position. Some models reuse the embedding table as this last layer (weight tying), which saves parameters - What the model outputs, precisely: not a token but a probability for every token in the vocabulary, at every position (97 numbers for our characters, about 50,000 for GPT-2's word pieces), summing to 1. In training nothing is sampled: the loss only reads off the probability given to the true next token. Only when the model writes text is one token drawn from these probabilities (or the most likely one taken, greedy decoding), appended, and fed back in; the slide "Generating text: sample, append, repeat" shows the loop

- Why several heads: one head produces one set of weights per token, so it can follow only one pattern at a time. The GPT-2 slides showed different heads doing different jobs (the previous token, the token after an earlier copy, the first token); several heads side by side let one block do several of these at once - The sizes, from the start. d_model is the width of the vector that represents each token everywhere in the model: the embedding, the residual stream, the output of every block. In the one-head model of the last lecture (the first part of the assignment), d_model = 128, and the head's query, key and value matrices were 128 × 128, so each query, key and value had 128 numbers. A query must have as many numbers as a key, because a score is their dot product, so one symbol, d_k, gives the length of both; d_v gives the length of a value. In practice d_v = d_k too - Splitting the width: with n_h heads, each head h has its own three matrices W_Q^(h), W_K^(h), W_V^(h), that map d_model numbers to d_model / n_h. In the model you train in the assignment, d_model = 256 and n_h = 4, so d_k = d_v = 256 / 4 = 64: each head's queries, keys and values have 64 numbers. Each head computes its own scores, its own weights a_ij and its own output Y^(h), a T × 64 table, exactly as on slide 2 - Putting them back together: the four outputs are placed side by side (concatenated), giving T × 256 again, and one more matrix W_O (256 × 256) mixes them. The result has the same shape as the input, so it can be added back to the residual stream - Why split the width rather than give every head the full width: each token's vector has d_model = 256 numbers (dimensions). Instead of one head whose queries, keys and values have 256 numbers each, we use four heads whose queries, keys and values have 64 numbers each. The four heads together have the same number of weights in W_Q, W_K and W_V as the single wide head, so the cost is about the same, but the block now computes four separate sets of attention weights - The head index (h) is a superscript in parentheses, as on the course notation sheet

- The symbols: T is the number of tokens in the window the model reads at once (up to 256 in the model you train, 1,024 in GPT-2); d_model is the number of dimensions of each token's vector (256 in your model); with 4 heads, each head's queries, keys and values have d_k = d_v = 256 / 4 = 64 numbers. The column "In your model" puts those numbers in - B, the batch: training does not feed one window at a time but B windows together (for example 64), because a GPU processes many windows in parallel as cheaply as one. In code, every shape has a batch dimension in front. In the model you train: (B, T, 256) in, four heads of (B, T, 64), concatenated back to (B, T, 256), then the output projection W_O. Setting one head's output to zero before the concatenation switches that head off - Checking shapes is the fastest way to debug attention code: every product above must line up, (T × d_k)(d_k × T) = T × T, then (T × T)(T × d_v) = T × d_v - The output has the same shape as the input, which is what lets blocks be stacked

- Where this sits: every block has two parts. Attention moves information between positions; this small network, an MLP (multi-layer perceptron, the kind of network of the lectures on learning weights), then processes each position's vector on its own. The same weights are used at every position: row 1, row 2, up to row T of the window go through the same network, and positions do not interact here - The formula, step by step, for one position t: the vector x, with d_model numbers, is multiplied by W^[1] and shifted by b^[1], giving the hidden layer; GELU is applied to each hidden number separately; the result is multiplied by W^[2] and shifted by b^[2], giving back d_model numbers. That output is added to the residual stream (next slide) - What "4 × wider" means, in the figure: in the model you train, d_model = 256, so x has 256 numbers. W^[1] maps 256 numbers to 1,024, so the hidden layer has 1,024 numbers, four times as many. W^[2] maps the 1,024 back to 256. The widening gives the network many units to combine features in; the narrowing returns to the width every other part of the model expects. In GPT-2 small the same ratio gives 768 → 3,072 → 768 - GELU stands for Gaussian Error Linear Unit (Hendrycks & Gimpel, 2016). GELU(z) = z · Φ(z), where Φ is the cumulative distribution of a standard normal: the probability that a normal random number is below z. So the input is multiplied by a number between 0 and 1 that rises smoothly with z: for large positive z, Φ(z) ≈ 1 and GELU(z) ≈ z; for large negative z, Φ(z) ≈ 0 and GELU(z) ≈ 0. It is a smooth version of the ReLU of the model-neuron lecture, with a small dip below zero around z = −1 (GELU(−1) ≈ −0.16). The smooth curve has a gradient everywhere, which helps training slightly; for intuition, think of it as a ReLU - Is GELU better only for transformers? No; the choice matters little in either. Hendrycks & Gimpel (2016) reported small gains over ReLU on both image and text tasks. It became the default in transformers mostly by inheritance: BERT and GPT-2 used it, and later models copied them. In CNNs, ReLU stayed standard (ResNet, the networks of the vision lectures); ConvNeXt (Liu et al., 2022), a CNN modernized to match vision transformers, switched to GELU and reported the same accuracy as with ReLU. Current large language models mostly use yet another variant, a gated unit called SwiGLU (Shazeer, 2020), which gives a small but consistent improvement. The nonlinearity is a detail; what matters is that there is one - Why it matters: the two matrices of the MLP hold most of a transformer's weights. In GPT-2 small (d_model = 768), W^[1] and W^[2] hold about 4.7 million weights per block, twice as many as the attention projections. It is the part that grows most in bigger models, and much of what a model stores about the world, such as facts, is thought to live here

- How to read the figure, bottom to top: the vector x of each position arrives from the block below and travels straight up the left side, the residual stream. Twice it branches off to the right. The first branch normalizes it (LayerNorm), runs multi-head attention, and the result is added back to the stream at the first plus sign. The second branch normalizes the new stream, runs the small network (MLP), and adds its result at the second plus sign. What leaves at the top goes to the next block. In formulas: X ← X + MHA(LN(X)), then X ← X + MLP(LN(X)) - So yes, there is a residual around the MLP as well as around attention: each sublayer's output is added to what came in, never substituted for it. If a sublayer outputs zeros, the stream passes through unchanged; that is also what a head knockout relies on - Where LayerNorm sits: on the branch, before each sublayer (pre-norm). The stream itself is never normalized inside the block; a final LayerNorm is applied once, after the last block, before the readout - One block is exactly this figure; the model stacks L of them, each with its own weights. The model you train has L = 4 blocks of width d_model = 256 with 4 heads each; GPT-2 small has 12 blocks of width 768 with 12 heads each - LayerNorm, concretely: take a position's 256 numbers, subtract their mean, divide by their standard deviation, so they have mean 0 and variance 1; then multiply each by a learned scale and add a learned shift. It keeps the numbers entering attention and the MLP in a stable range however large the stream grows - You met residual connections in the visual-system lecture (ResNet-50's block groups): the addition gives gradients a direct path through many layers - LayerNorm is a cousin of the batch normalization from that lecture. BatchNorm normalizes each channel across the images of a batch; LayerNorm normalizes each position across its own d_model numbers, so it does not depend on the batch - The original transformer normalized after adding (post-norm): X ← LN(X + Sublayer(X)). Current models normalize before each sublayer (pre-norm), which Xiong et al. (2020) showed keeps gradients well behaved at the start of training, without a warm-up of the learning rate - Information and gradients travel along a sum, so a later block can read what an early head wrote. The gated recurrent networks of a later lecture carry their memory in a similar way, partly by addition

- Why this slide comes here, after the residual slide and not with multi-head attention: heads in the same block run side by side on the same input and cannot use each other's results. Here one head must use what another head has already written. That needs two blocks in a row: block 1 writes into the residual stream, block 2 reads it. Several heads in one block let that block read in several ways at once; several blocks in a row let a later head build on an earlier one. Copying needs the second - The order to explain it, following the numbered steps. The sentence is The black cat sat on the mat. The black, and the model should predict cat next - Step 1, the problem, the faint dashed arrow. Suppose the model had a single attention layer, like the one-head model of the last lecture. At the second black, the head can look for an earlier token like itself and find the first black. But reading the first black gives back black, which says nothing about what comes next. What it needs is the token after the first black, and nothing in a single layer points there. This is why the one-head model you build first could not copy a new name - Step 2, block 1, the top band. A previous-token head: each token after the first reads the token just before it and adds what it read to its own vector in the residual stream. Afterwards, the first cat's vector carries a note, "the token before me was black". That is the head of slide 6 - Step 3, block 2, the bottom band. An induction head at the second black. Its query asks: which earlier position says "the token before me was black"? The key of the first cat, built from its vector, which now holds that note, matches: the gold arrow. The head reads that position's value, cat, and the readout turns it into the prediction cat. That is the head of slide 7 - The two heads compose: the second could not find the first cat without the note the first wrote. Elhage et al. (2021) worked this out in two-layer attention-only models, and Olsson et al. (2022) found it in large ones - In GPT-2, block 4, head 11 is the strongest previous-token head and block 5, head 5 the strongest induction head, one block later; both can be opened in the head explorer. In the assignment you look for the same pair in the model you train

- For home: this demo goes through one block one step at a time; in class we only point to it. Read these notes alongside it. - What the demo is: the block of the residual slide, animated on three tokens. It is a schematic, with no trained numbers. The two buttons at the top switch between "Transformer overview" (this view) and "Attention, step by step" (one head with exact arithmetic, the demo of the last lecture). Back, Play, Next and Reset step through six stages; the tabs across the top jump to any stage. The box on the left always shows where you are in the stack: embedding, attention, add, small network, add, repeat, readout - Stage 1, Embed: three tokens, the, cat and sat, each a vector, sit side by side, one vertical stream per position. The stream is the residual stream of the residual slide: everything that follows reads from it and adds to it - Stage 2, Attention: the communication step, the only one in which positions exchange information. A normalized copy of each stream (LayerNorm) produces a query, a key and a value; each position reads itself and the positions before it. The button "Zoom into attention" opens the step-by-step view of one head. Several heads do this in parallel; the demo follows one - Stage 3, Add back: the attention result is multiplied by W_O, which brings it back to the stream's width, and added to the stream. The bypass line around the attention box is the residual connection: h = x + update. The original vector is kept, and attention only adds to it - Stage 4, Small network: a normalized copy of each stream goes through the MLP, separately at each position with the same weights. Positions do not talk to each other here: the network combines features within one vector - Stage 5, Add again and repeat: the MLP's output is added back to the stream, which finishes one block. The next block takes this stream as input and recomputes its own queries, keys and values, with its own weights. The model you train repeats this 4 times, GPT-2 small 12 times - Stage 6, Predict: after the last block, a readout (a linear layer and a softmax) turns each position's vector into probabilities over the vocabulary. Two different softmaxes: attention's softmax is over positions to read; the output softmax is over tokens to predict. In training every position predicts the token that follows it; when writing text, only the last position is used - What to point out while clicking: only stage 2 moves information between columns; every other stage works on each column alone. And at stages 3 and 5 the stream is added to, never replaced

- In the figure, an underscore stands for a space; each position's target is the input one step later - This is the cross-entropy from the lecture on learning the weights, with one classification per position: the classes are the characters of the vocabulary - Nats: the loss uses the natural log. A model that guesses uniformly over K characters (the classes here) scores ln K; lower is better - A training window therefore holds T + 1 characters: the first T are the input, the last T the targets - Feeding the true text as input, rather than the model's own guesses, is called teacher forcing - GPT-2, earlier today, used this same objective with word pieces; a later lecture scales it to billions of parameters

- AdamW is Adam (from the backpropagation lecture, the optimizer of the earlier assignments) with weight decay applied directly to the weights rather than through the gradient; it is the default optimizer for transformers - Perplexity and loss are the same number in two currencies: a loss of 1 nat per character is a perplexity of e ≈ 2.7, as if the model chose uniformly among 2.7 characters at every step. Papers on large models report either - PyTorch's cross_entropy expects the classes in the second dimension, so the (B, T, K) logits are reshaped to (B·T, K) and the targets to (B·T) - The training loss on random windows and the held-out loss should fall together. When the training loss keeps falling and the held-out loss stops, the model has started to memorize its training text, the overfitting of the generalization lecture - A first check on any new model: the loss at step 0 should be close to ln K. Much higher means the initial weights already make confident, wrong guesses

- Generation uses the model one position at a time, but training used all positions at once. The causal mask makes the two consistent: during training, each position saw only what it would see at generation time - The scores here are the logits, the numbers the last layer produces before the softmax - Temperature, concretely: the softmax turns logits z into probabilities proportional to e^z. Dividing every logit by a temperature before the softmax changes how peaked the result is, without changing the order of the tokens. Example with two tokens whose logits are 2 and 1: at temperature 1 their probabilities are 0.73 and 0.27; at 0.5 (logits 4 and 2), 0.88 and 0.12; at 2 (logits 1 and 0.5), 0.62 and 0.38 - Predictable versus "creative": at a low temperature the model almost always picks its top choices, so the text is safe, repetitive, and the same every time; at temperature near 0 it is greedy and often loops. At a high temperature unlikely tokens are drawn more often, so the text surprises more; push it further and the rare tokens are nonsense. "Creative" here only means less probable under the model, not better ideas - The next slide shows the same prompt continued by GPT-2 at four settings; the demo in "Going further" lets you move the temperature yourself

- The four continuations come from the same pretrained GPT-2 small (124 million parameters, 2019) with the same random seed; only the temperature differs. Nothing was selected: these are the first draws. The entropy in nats of the next-piece distribution at the first step, exponentiated, gives the effective number of word pieces the draw chooses among: 7 at temperature 0.7, 42 at temperature 1 (the model's own distribution), 2,630 at temperature 1.5 - Greedy decoding (temperature 0) repeats itself; Holtzman et al. (2020) showed that beam search and greedy decoding loop even in GPT-2 large, while sampling from the full distribution drifts into nonsense because the long tail of unlikely pieces is sampled too often. Their fix, nucleus sampling, keeps only the smallest set of pieces whose probabilities add up to p (say 0.9) and renormalizes - The temperature setting of a chat system's API is exactly the temperature of the slide "Generating text: sample, append, repeat". Kempton and Burrell (2025) analyze what the renormalization inside top-k, nucleus and temperature sampling does to the distribution the model actually learned

- A trained model is a system we can record from and lesion, as a neuroscientist would: every unit and every weight is available

- A readout on a model layer is the same tool you used on recorded neurons: a linear classifier fitted on one part of the data and scored on held-out data. Alain and Bengio (2016) called such classifiers "probes" and fitted one after every layer of a network - A readout tells you what a layer holds in a linearly usable form, not whether the model uses it. That is why the next slides lesion and patch - Two more cautions: a readout that does worse at a later layer does not show that the information is gone (it may be there in a form a line cannot read), and a readout that agrees with what a head seems to do does not show that the head is responsible for it - Csordás, Manning and Potts (2025) used readouts and skipped blocks on Llama 3.1, Qwen 3 and OLMo 2: the second half of the blocks adds much less to the residual stream than the first, and skipping them costs little - A trained model is a system we can record from and lesion, as a neuroscientist would, except that every unit and every weight is available

- Clark et al. (2019) read BERT's 144 heads one by one; some single heads found the direct object of a verb, or the noun a determiner belongs to, with high accuracy - Voita et al. (2019) switched heads off in a translation model: most could be removed with little loss, and the few that mattered had clear roles (previous word, rare words, syntactic relations) - Why weights are not explanations: a head can put large weight on a position whose value vector is close to zero, so it contributes little; and later layers can undo or override what a head wrote

- This is the name test of the assignment. The loss is measured only on the letters of the second mention after its first letter, each predicted from the true text before it, so the model is not writing the name freely - Comparing with a control story separates copying from a general change in how well the model predicts letters: a lesion that makes every prediction worse raises both losses. The benefit then changes only if the two losses rise by different amounts, so compare the two losses and the ordinary-text loss as well as their difference - Why the Molly story is the control: everything is the same except the first name. In the Zefam story, when the model reaches the second Zefam and has seen its first letter Z, an earlier Zefam is in the context, so a head can find it and read the letters that followed: e, then f, a, m. In the Molly story the second name is also Zefam, but the only earlier name is Molly. There is no earlier Zefam to find, so the model can only guess e, f, a, m from what it knows about spelling in general, and its loss on those letters is high. The difference between the two losses is what copying from the first mention buys: the benefit - Molly is a simple control with one flaw: it is a name the model saw in training

- Olsson et al. (2022) knocked out induction heads in small attention-only models and in-context learning, the drop in loss late in a text, largely disappeared - Voita et al. (2019) removed most heads of a translation model with little loss: many heads are redundant, so a single knockout can understate what a group of heads does together - A knockout tests whether a head is needed, not what it computes. A head can matter only in combination with others, and the rest of the network can partly compensate, as the brain sometimes does after a lesion - Neuroscience does the same with lesions, cooling and optogenetic silencing - What setting a head's output to zero does, concretely. Inside a block, each head h computes its output Y^(h), a T × 64 table (one 64-number vector per token: that head's weighted average of values). The four tables are placed side by side into a T × 256 table and multiplied by W_O, and the result is added to the residual stream. Multiplying by W_O is the same as giving each head its own 64 rows of W_O and adding the four contributions: attention output = Σ_h Y^(h) W_Y^(h). A knockout replaces Y^(h) of one head by zeros, so its term in that sum is zero: whatever that head read is never written into the residual stream. Everything else is unchanged: the other heads still write their contributions, the token's own vector still flows through the residual connection, and every later block runs as usual on what remains. The model then makes its predictions without that head, and we measure the loss again. In code, it is one line: multiply that head's output by 0 before the heads are concatenated (the assignment's head_scale switch) - Zeroing a head's output is the simplest knockout; replacing it with its average output over many inputs (mean ablation) keeps the downstream layers in a more familiar range - The next slide runs this experiment on a real model

- The experiment of the previous slide on a pretrained model. Each of GPT-2 small's 144 heads gets an induction weight: on 50 random word pieces repeated twice, the average weight a position in the second copy puts on the piece that followed its earlier copy. The top 8 heads reach 0.50 to 0.93; the median head 0.005. Nothing about the heads was chosen by hand - Left: the loss of predicting the second copy of a fresh repeated random sequence, which can only be done by copying. Intact, 0.25 nats per piece, a geometric-mean probability of about 0.78 for the correct next piece. With the 8 induction heads zeroed, 7.49: copying is severely impaired (a uniform guess would be 10.8). With 8 random other heads zeroed (10 draws), 0.81 on average; the range bar spans the 10 draws, up to 2.7, because heads just below the cut also copy a little - Right: 20 ordinary English sentences, the control behaviour. The same lesion costs 0.56 nats, the random one 0.15. The induction heads do matter a little for ordinary text, where repeated words are common, but far less than for copying - Wu et al. (2025) ran the same logic in a 7-billion-parameter model with 80,000-token contexts: masking the 20 heads that copy from the context makes it confabulate answers about a sentence hidden in the context, masking 20 random heads does nothing (the optional slide "Twenty heads find a fact in a long document") - The assignment runs this on the model you train, one head at a time, with the same controls. You do not knock out GPT-2's heads yourself: in the assignment GPT-2 appears only at the end, to compare its word-by-word surprisal with human reading times

- The figure is the patch of the assignment, schematic. The method comes from Meng et al. (2022), "causal tracing" in GPT-2 XL, their Figure 1: (a) The clean run reads "The Space Needle is in downtown" and predicts "Seattle". (b) The corrupted run adds noise to the embeddings of the subject tokens, and the prediction is lost. (c) One hidden state at one token and one layer is copied from the clean run into the corrupted run; (d) the probability of "Seattle" is read out again. (e)–(g) That probability for every token and layer: restoring a single hidden state works at an early site (the last subject token, middle layers) and a late site (the last token, late layers); restoring MLP outputs recovers the early site, restoring attention outputs the late site. Their reading: the MLPs of the middle layers store the fact, attention at the end moves it to where it is needed - The logic is the converse of a lesion: a lesion removes a part and shows what fails; a patch restores a part and shows what returns. Neuroscience has the same pair, inactivation versus stimulation or replay - The recovery is the fraction of the clean-minus-corrupted gap that the patch closes. Zhang and Nanda (2024) review the choices that change the answer: which metric (probability, logit difference, loss), what to corrupt (noise or a different input), and how many components to patch at once. Two controls make a recovery mean something: a donor from a different clean input, and a patch of a part that the knockout said did not matter - In the assignment the donor is the clean story with the same name and the corrupted story carries a misleading name; the patch replaces one head's output at the positions that predict the name's letters

- Long range: in one attention step, a token can read any earlier token directly, however far back. Content-dependent: the weights are computed from the input itself, query against key, so which tokens are linked changes with every input. The usual terms are data-dependent or dynamic weights, and content-based addressing: a token finds what to read by what it contains, not by where it sits. Because a score multiplies a query by a key, both computed from the input, attention is a multiplicative interaction between tokens, unlike a weighted sum with fixed weights. A convolution, in the visual-system lecture, had fixed weights for each relative position; a recurrent network, next lecture, passes information along one step at a time - Fast to train: because of the causal mask, one pass over a window gives a prediction, and a loss, at every position at once; no position has to wait for the one before it. That is why transformers could be trained on far more text than recurrent networks. Vaswani et al. (2017), "Attention is all you need", built a translation model from attention and small per-position networks only; it set the best English-to-German score of its time after 3.5 days on 8 GPUs - Scale, in words: one GPT-2 token is about three-quarters of an English word on average. GPT-2 small's window of 1,024 tokens is about 750 words, two pages of a book. A window of a million tokens is about 750,000 words: roughly 2,500 book pages, or eight novels of 90,000 words. GPT-2 was trained on WebText, about 40 GB of text from 8 million web pages, roughly 9 billion tokens; current models are trained on more than 10 trillion tokens, a thousand times more - The price. T is the number of tokens in the window. Every token's query is compared with every token's key, so the score table has T × T entries: twice the length, four times the work. And to write the next token, the model keeps the keys and values of every earlier token in memory, for every block and head: the "KV cache". Long contexts are therefore expensive in memory as well as in compute, and much current research looks for cheaper attention, including the recurrent models of a later lecture - A long window does not mean the model uses all of it. Liu et al. (2024) placed the answer to a question at different positions of a long context: accuracy is highest when it is at the start or the end and sags when it is in the middle, even for models built for long inputs

- Next lecture: word pieces, scale, in-context learning, and language models as models of human reading and of the brain. The lecture after: a different way to carry context, recurrence, and why it is coming back

- Submitted on Canvas (Minute papers → Minute Paper 17); credit for a thoughtful attempt

- Yin and Steinhardt (2025) and Feucht et al. (2025) both revise the induction-head story: the first finds that function-vector heads, not induction heads, carry few-shot learning in large models; the second finds a parallel route of induction heads that copy whole lexical units, responsible for translation-like tasks, so that ablating token-level induction heads makes a model paraphrase instead of quoting - Aoyama, Wilcox and Schneider (2025) predict the training step at which induction heads form from batch size and context length, independently of model size, and show which statistics of the data (repeated bigrams) are needed for them to form at all - Csordás, Manning and Potts (2025) measure how much each block of Llama 3.1, Qwen 3 and OLMo 2 adds to the residual stream: the second half of the blocks contributes much less, skipping them costs little, and deeper models seem to spread the same computation over more blocks rather than compose new steps - Khandelwal and Pavlick (2025), from Brown, ask how a model answers a two-step question such as the capital of the country where a river is: decoding the residual stream shows two mechanisms, one that computes the intermediate answer on the way and one that jumps directly, and the embedding geometry predicts which is used - Méloux et al. (2025) is the caution for the whole programme: in networks small enough to enumerate every candidate, several circuits reproduce the same behaviour and several algorithms fit the same circuit - Also worth reading: Bick, Xing and Gu (2026, arXiv:2602.11374) on how 2% of heads recover 95% of in-context retrieval; Zukowska et al. (2026, arXiv:2604.14477) on circuits in vision transformers; Izadi et al. (2026, arXiv:2603.13652) on activation patching; Oh et al. (Trends in Cognitive Sciences 2026, doi:10.1016/j.tics.2026.05.008) on making language models better models of human reading; Zhang et al. (eLife) on feature attention during free-gaze search in V4 and IT - More that qualify: Olsson et al. (2022, arXiv:2209.11895) and Reddy (ICLR 2024, arXiv:2312.03002) on induction heads; Voita et al. (ACL 2019, arXiv:1905.09418) and Clark et al. (BlackboxNLP 2019, arXiv:1906.04341) on head pruning and head roles; Minegishi et al. (ICML 2025, arXiv:2505.16694) on circuits that emerge in phases when models must infer a task from examples; Saanum et al. (BlackboxNLP 2025, arXiv:2509.21534) on a circuit for hierarchical patterns in context; Yang et al. (ICML 2025, arXiv:2502.20332) on symbolic induction heads for abstract reasoning; Sharkey et al. (2025, arXiv:2501.16496) on open problems in mechanistic interpretability; Mueller et al. (ICML 2025, arXiv:2504.13151), a benchmark for circuit-finding methods; Somvanshi et al. (ACM Computing Surveys 2026, doi:10.1145/3787104), a survey; Bick, Xing and Gu (2026, arXiv:2602.11374): keeping 2% of the heads recovers 95% of retrieval when the rest are made recurrent; Cottier et al. (2024, arXiv:2405.21015) on training costs; Hong et al. (eLife 2026) and Cai et al. (Nature 2026) on language models and the brain - Before posting, skim the existing entries on your paper; your post must add something not already said

- Why this slide: slides 6 to 9 looked at one short sentence in a small model. Here the same kind of head matters in a large model reading a long document, and the evidence is a knockout, the method of the demo's knockout panel - The model: LLaMA-2 with 7 billion weights, 32 blocks of 32 heads (1,024 heads), in a version extended by further training to read up to 80,000 tokens (LLaMA-2-7B-80K). It was trained, like GPT-2, only to predict the next token - The test, "a needle in a haystack": a long filler document (essays) with one unrelated sentence hidden in it, "The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day". At the end the model is asked "What is the best thing to do in San Francisco?" To answer, it has to find that sentence and reproduce it - How to read the figure: each grid is a set of tests. Across: the length of the document, from 5,000 to 100,000 tokens. Down: where the sentence was hidden, from 10% of the way in (top) to 90% (bottom). Each cell is one test, colored by how much of the answer is right: green correct, yellow partly, red wrong. The number above each grid is the average over all cells - How retrieval heads were found: while the model writes its answer, each word it writes that also appears in the hidden sentence is a copied word. For every head, Wu et al. check whether its largest attention weight at that moment falls on that same word in the hidden sentence. The fraction of copied words for which it does is the head's retrieval rate. Few heads have a high retrieval rate: under 5% of all heads, in every model family they tested - The result: masking the 20 heads with the highest retrieval rate (setting their attention so they read nothing) drops accuracy to 63.6% (left grid, many yellow and red cells), while masking 20 heads chosen at random leaves it at 94.7% (right grid, almost all green). Twenty heads out of 1,024, about 2%, carry much of the ability. When they are masked, the model often still answers fluently but makes something up - Why accuracy does not fall to zero: other heads copy a little too, and the model can sometimes guess, as with the induction heads of GPT-2, where several heads share the job - The link to today: a retrieval head does what an induction head does at long range, reading a position earlier in the text and bringing its content forward, though the two are found with different tests. The design is the assignment's: knock heads out, measure what the model can no longer do, and compare with a control set of heads - Why it matters in practice: methods that shrink the memory a long-context model needs (the keys and values of every earlier token, the KV cache of the slide "What attention gains, and what it costs") work better when they keep the full context for the retrieval heads

- You saw this figure in the lecture on learning without labels. Here the point is the mechanism: it is the same attention as in GPT-2, with image patches as the tokens instead of words - How an image becomes tokens: the image is cut into a grid of 8 × 8-pixel squares, the patches. Each patch's 192 numbers (8 × 8 pixels × 3 colors) are mapped by a learned matrix to a vector, the patch's embedding, the way a word's token is looked up in a table; a position embedding says where the patch sits. One extra learned token, the class token, is added to the patches; its final vector is the summary of the whole image that the training uses - No causal mask: an image has no "later", so every token can read every other token in both directions - How to read each pair: the photograph, then the attention map. The map shows the class token's row of the weight table A: one weight a_ij for each patch j, from the heads of the last block, drawn at that patch's position. Bright means a large weight. Each row of A still sums to 1, spread over all the patches - Why the maps look so fine: the patches are small. This model (ViT-S/8) uses 8 × 8-pixel patches, and the maps are computed on images several hundred pixels wide; at 480 × 480 pixels that is 60 × 60 = 3,600 patches, so each map has 3,600 cells. Zoom in on a map and the small squares are visible. A more common vision transformer with 16 × 16-pixel patches on a 224-pixel image has only 14 × 14 = 196 patches and much coarser maps - Why it outlines the object: DINO was trained without labels, by making the class token's output agree across two different crops of the same photograph. To do that, the class token learns to read the patches that identify the image, which are the patches of the main object, and to ignore the background. Nobody gave the model a label, a mask or a box - The same architecture, scaled up, is behind current image and video models. The next slide shows that vision transformers also have attention sinks, and a later lecture returns to how well such models match the visual system

- Darcet et al. (2024) found that large vision transformers produce a few tokens with enormous activations, on background patches that carry little information about the image. Other heads put most of their weight on those tokens: they are attention sinks, doing the job of GPT-2's first token, a place to park weight. Because the class token's weight goes there too, its attention map hides the object - Jiang et al. (2025) traced the enormous activations to a small set of neurons in the small network (MLP) of one block, which they call register neurons. A neuron's output is one number at each patch position. For these neurons the number is near zero at most patches and huge at a few, and those few patches become the sinks - What "moving the activations" means concretely, and where it happens: not on the image, whose pixels are never changed, but on the tokens inside the network, the vector each patch has at that block. Run the trained model on an image, and when the computation reaches that block, overwrite the outputs of those few neurons. At the patches where they were huge, set them to zero; at the patches you choose, write the huge value instead. The model then continues its computation as usual from the edited numbers. No weight is changed and nothing is retrained. This is the same kind of intervention as the assignment's patching, where a head's output is replaced during a run - If the bright patches appear wherever the edit puts the huge values, the neurons are the cause. The next slide shows that they do: the authors plant a smiley face and the sinks draw it - A register is an extra token that is not part of the image. Darcet et al. added such tokens during training and the artifacts disappeared; Jiang et al. add one to an already trained model and move the huge values into it, with the same effect

- Two large vision transformers, OpenCLIP (trained on images paired with captions) and DINOv2 (trained without labels), each on two photographs, an eagle and a dog. Each map is the class token's attention over the patches, bright where its weight is large, as on the DINO slide - Original: the model as trained. A few isolated bright squares on the sky or the grass take most of the weight; the animal is hard to see. Those squares are the sinks of the previous slide - Shifted: the same model and photograph, with the register neurons' outputs edited as on the previous slide. The smiley face and the heart are planted by the authors: they picked target patches in those shapes and wrote the huge values there. The sinks then appear exactly on those patches and nowhere else. That is the test: if editing a few neurons decides where the sinks appear, those neurons are their cause, and nothing in the image is - With a test-time register: one extra token added to the input, and the huge values moved into it. The extra token becomes the sink, and the class token's map over the image patches now outlines the eagle and the dog - None of the three columns involves retraining. The point for the course: find the few components behind a behavior, intervene on them, and check that the behavior moves. The assignment's knockouts and patches are a small version of this approach

- The assignment's model has a 97-character vocabulary and a few million weights. The next slides run two of the same pieces, sampling and the held-out loss, on pretrained models, from GPT-2 to a model trained on children's stories

- Move the temperature: below 1 the top pieces take almost all the probability; above 1 the long tail grows, and the number of effective choices (e to the entropy of this one distribution) climbs from about 7 at temperature 0.7 to 2,630 at temperature 1.5 - Move top-p below 1: only the most likely pieces that together reach p are kept, and their probabilities are rescaled to add up to 1. This is how chat systems cut the tail - Draw a few pieces at temperature 1.5 and see how often the draw lands far down the list - The right panel shows what GPT-2 writes at the nearest precomputed setting. In the assignment, you sample from your own model at temperatures 0.5 and 1.5

- The bigram table scores 2.30 here and 2.28 in the table at the start of the lecture: the same table, scored on a different held-out sample (40 whole stories here, random windows there) - Every bar is the cross-entropy of the same text, the first 40 stories of the TinyStories validation split (Eldan & Li 2023), divided by the number of characters predicted. Word-piece models predict several characters at once, so their loss per piece is divided by the characters those pieces cover: 2.40 nats per piece for GPT-2 small becomes 0.60 nats per character - The bigram table and the 4-block model are the ones of the assignment (97 characters, 256 positions of context, 3.3 million weights, 15,000 AdamW steps). GPT-2 small, medium and large (Radford et al. 2019) were trained on 40 GB of web text and have never seen these stories; they are scored as they are, with 1,024 word pieces of context - The comparison is on children's stories, a narrow kind of text, which is why a tiny specialist can match a general model. TinyStories-33M was trained by Eldan and Li on the training split of the same collection, so for it this text is in distribution. Which model is "better" depends on the text you score it on: the lecture on baselines made the same point

- Yin and Steinhardt (2025) compared the two mechanisms proposed for in-context learning across 12 models from 70 million to 6.9 billion parameters. Induction heads copy; function-vector heads (Todd et al. 2024) carry a compact code of the task shown by the examples ("antonyms", "capital of"). Ablating the FV heads, a few percent of all heads, destroys few-shot accuracy; ablating as many induction heads barely changes it, especially in larger models. Over training, many FV heads start out as induction heads and then change role, so induction may be the scaffold on which the more abstract mechanism is built - The figure is panel (a) of their Figure 1, for Pythia 6.9B: accuracy on few-shot tasks as a growing fraction of heads is removed, for random heads, induction heads and FV heads - Méloux et al. (2025) ask whether a mechanistic explanation is unique. In small networks where every candidate can be enumerated, several circuits reproduce the same behaviour and several algorithms fit the same circuit. A lesion or patch result tells you that a part is sufficient or necessary for a behaviour, not that you have found the only story - Lesion, patch, control, then test the story at the next scale: that is the method, in a model as in a brain