Building & probing a language model

Reader-friendly version of the 42 lecture slides: the lecture content in normal flow, with figure descriptions and data tables where the figure carries data. Open the presentation.

Slide 1

Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 16 Β· Theme 3: Dynamics, sequences & world models

Building & probing a language model

Thursday, November 12 Β· Fall 2026

Return to contents

Slide 2

Where we left off: one attention head

𝐘=softmax ⁣(𝐐𝐊⊀dk)⏟attention weights π€π•\mathbf{Y}=\underbrace{\mathrm{softmax}\!\left(\frac{\textcolor{#B5396B}{\mathbf{Q}}\textcolor{#3E6D8E}{\mathbf{K}}^\top}{\sqrt{d_k}}\right)}_{\text{attention weights }\mathbf{A}}\textcolor{#3F7A6B}{\mathbf{V}}

Indices: ii is the token doing the reading (a row of the tables), jj a token being read (a column)

  1. Scores sij=πͺi⋅𝐀js_{ij}=\textcolor{#B5396B}{\mathbf{q}_i}\cdot\textcolor{#3E6D8E}{\mathbf{k}_j}: how well token ii's query matches token jj's key, for every pair, divided by dk\sqrt{d_k}
  2. Causal mask on the score table: scores of later tokens (j>ij>i) become βˆ’βˆž-\infty, so their weights are 0
  3. Weights aija_{ij}: the softmax turns each row of scores into positive numbers that add up to 1; a larger score gets a larger weight
  4. Output 𝐲i=βˆ‘jaij 𝐯j\mathbf{y}_i=\sum_j a_{ij}\,\textcolor{#3F7A6B}{\mathbf{v}_j}: token ii's new vector, the values averaged with its weights

Return to contents

Slide 3

From the head's output to the loss

  1. Readout: logits 𝐳i=𝐖𝐲i\mathbf{z}_i=\mathbf{W}\mathbf{y}_i, one per character; probabilities 𝐩i=softmax(𝐳i)\mathbf{p}_i=\mathrm{softmax}(\mathbf{z}_i)
  2. Loss at each position ii: βˆ’ln⁑-\ln of the probability given to the true next character. The training loss averages these numbers over all positions
  3. Training lowers the loss by changing all the weights: where a head looks is learned

To write text, the trained model draws the next token from 𝐩i\mathbf{p}_i, appends it, and repeats (later today)

From the head's output to the loss β€” table
Model Loss (nats) Perplexity elosse^{\text{loss}}: equally likely characters
Uniform guess 4.57 97
Bigram table 2.28 9.8
Average of earlier characters 2.03 7.6
One attention head 1.23 3.4

Return to contents

Slide 4

Attention in real systems

Today: what trained heads do in real models; then build a whole model from heads, train it to write, and open it up

Return to contents

Slide 5

GPT-2: a language model we can open up

Return to contents

Slide 6

A previous-token head

Attention weights of GPT-2 small's block 4, head 11 on the sentence The black cat sat on the mat. The black cat sat on. Rows are the token being updated, columns the token being read; only the lower triangle is filled. A dark stripe runs just below the diagonal: every token after the first puts almost all its weight on the token just before it. Average weight on the token before: 0.999

Computed from GPT-2 small (124 million parameters, 144 heads). The model is given no task: it reads the sentence "The black cat sat on the mat. The black cat sat on" and, as in training, predicts the next token at every position, and we record one head's attention weights. A square matrix, rows the token being updated (the query), columns the token being read (the key), with the word pieces of the sentence along both axes; each cell is a softmax weight, after the causal mask, from near-white (0) to dark purple (1); every row sums to 1 and the upper right triangle is empty, because a token may read only itself and earlier tokens. Block 4, head 11, a previous-token head: a dark stripe just below the diagonal, each token reading the one before it. Printed below: average weight on the token before, 0.999. The head was chosen as the one with the largest such average on 20 separate sentences.

Open full-size figure

Real computation: GPT-2 small, weights as trained Β· head chosen on separate test inputs

  • The input: GPT-2 reads The black cat sat on the mat. The black cat sat on and predicts the next token at every position, as in training. We record one head's weights
  • Each row: one token's attention weights aija_{ij} over itself and earlier tokens; they sum to 1
  • This head: each token after the first reads the token just before it, so its output carries that token's value

Return to contents

Slide 7

An induction head

Attention weights of GPT-2 small's block 5, head 5 on the same sentence. In the first clause almost every row puts its weight on the first token. In the repeated clause the second black reads the first cat and the second cat reads the first sat, the tokens that followed their earlier copies. Average weight on the token after the earlier copy, over the repeated clause: 0.608

Computed from GPT-2 small (124 million parameters, 144 heads). The model is given no task: it reads the sentence "The black cat sat on the mat. The black cat sat on" and, as in training, predicts the next token at every position, and we record one head's attention weights. A square matrix, rows the token being updated (the query), columns the token being read (the key), with the word pieces of the sentence along both axes; each cell is a softmax weight, after the causal mask, from near-white (0) to dark purple (1); every row sums to 1 and the upper right triangle is empty, because a token may read only itself and earlier tokens. Block 5, head 5, an induction head. In the first clause no token has appeared before, so there is no earlier copy to find, and the weight goes to the first token. In the repeated clause, the second "black" reads the first "cat" and the second "cat" reads the first "sat", the tokens that followed their earlier copies. Printed below: average weight on the token after the earlier copy, over the repeated clause, 0.608. The head has the largest induction weight of all 144 heads on separate repeated random sequences (0.93).

Open full-size figure

Real computation: GPT-2 small, weights as trained Β· head chosen on separate test inputs

  • Text repeats itself: names, phrases. In a repeat, what followed a token last time is a good guess for what follows it now
  • This head reads exactly that: in the repeat, black reads cat, cat reads sat; in the first half it parks on the first token
  • Why "induction": the text showed A B once; when A returns, the head points to B, a rule induced from the context itself
  • Our one-head model could not do this: it takes two heads in a row (later today)

Return to contents

Slide 8

An attention sink

Attention weights of GPT-2 small's block 9, head 11 on the same sentence. One dark first column: almost every token, in both halves of the sentence, puts almost all its weight on the first token. Average weight on the first token: 0.926

Computed from GPT-2 small (124 million parameters, 144 heads). The model is given no task: it reads the sentence "The black cat sat on the mat. The black cat sat on" and, as in training, predicts the next token at every position, and we record one head's attention weights. A square matrix, rows the token being updated (the query), columns the token being read (the key), with the word pieces of the sentence along both axes; each cell is a softmax weight, after the causal mask, from near-white (0) to dark purple (1); every row sums to 1 and the upper right triangle is empty, because a token may read only itself and earlier tokens. Block 9, head 11, an attention sink: one dark first column, almost every token in both halves of the sentence putting almost all its weight on the first token. Printed below: average weight on the first token, 0.926. The head is the strongest sink (0.983 on 20 separate sentences) among heads that do not copy.

Open full-size figure

Real computation: GPT-2 small, weights as trained Β· head chosen on separate test inputs

  • This head: almost every token puts almost all its weight on the first token, in both halves
  • Why: a row's weights must add up to 1, so a head with nothing useful to read still puts them somewhere
  • What it does: its output is then the first token's value, nearly the same everywhere: the head contributes almost nothing new
  • Vision transformers do the same, on background patches; spare "register" tokens give the weight a place to go (Going further)

Return to contents

Slide 9

Try it: all 144 heads of GPT-2

Return to contents

Slide 10

From a head to a model

Attention moves information between positions. What else does a language model need?

Return to contents

Slide 11

The whole model, from the last lecture

Bottom to top, one column per token:

  1. Embedding + position: token β†’ vector
  2. Attention (gold): mixes positions
  3. MLP: each position alone
  4. 2 and 3 repeat in LL blocks
  5. Readout: a probability for every possible next token

Next: several heads, the MLP, the wiring

Schematic of a transformer, read from the bottom up, with one column per token. At the bottom, six tokens, the characters t, h, e, a space drawn as an underscore, c and a. Each becomes a small blue vector, labelled embedding plus position. A grey line runs up from each vector, the token's residual stream. Inside a dashed frame marked times L blocks, a gold self-attention band spans all six columns: lines from every column fan in to the last one, so the last token reads itself and the five earlier tokens and none after it. Above it, each column has its own small box labelled MLP, the small network that works on each token alone. At the top, an output layer and softmax spans the columns and gives each column's predicted next token: h, e, space, c, a, t, the input shifted by one

Schematic. A decoder-only transformer read from the bottom up, with one column per token. At the bottom, six input characters: t, h, e, a space drawn as an underscore, c and a. Each becomes a small blue vector labelled "embedding + position". A grey vertical line runs up from each vector: that token's residual stream, to which every later step adds its result. Inside a dashed frame marked "times L blocks", a gold self-attention band spans all six columns; lines from every column fan in to the last one, so the last token reads itself and the five earlier tokens and none after it. Above it, each column has its own small box labelled MLP, a small network that works on each token alone. At the top, an "output layer + softmax" band spans the columns and gives each column's predicted next character: h, e, space, c, a, t, the input shifted by one place. The model in the assignment has 4 blocks, width 256 and 4 heads per block.

Open full-size figure

Return to contents

Slide 12

Multi-head attention

Several heads let one block read in several ways at once. Run nhn_{\mathrm{h}} heads side by side:

𝐘(h)=Attention ⁣(𝐗𝐖Q(h), 𝐗𝐖K(h), 𝐗𝐖V(h))\mathbf{Y}^{(h)}=\mathrm{Attention}\!\big(\mathbf{X}\mathbf{W}_Q^{(h)},\,\mathbf{X}\mathbf{W}_K^{(h)},\,\mathbf{X}\mathbf{W}_V^{(h)}\big)

MHA(𝐗)=Concat(𝐘(1),…,𝐘(nh)) 𝐖O\mathrm{MHA}(\mathbf{X})=\mathrm{Concat}\big(\mathbf{Y}^{(1)},\dots,\mathbf{Y}^{(n_{\mathrm{h}})}\big)\,\mathbf{W}_O

Return to contents

Slide 13

Shapes at a glance

Shapes at a glance β€” table
Quantity Shape In your model One row is
𝐗\mathbf{X} TΓ—dmodelT\times d_{\text{model}} TΓ—256T\times 256 one position's vector
𝐐, πŠ\textcolor{#B5396B}{\mathbf{Q}},\ \textcolor{#3E6D8E}{\mathbf{K}} (one head) TΓ—dkT\times d_k TΓ—64T\times 64 one query, one key
𝐕\textcolor{#3F7A6B}{\mathbf{V}} (one head) TΓ—dvT\times d_v TΓ—64T\times 64 one value
scores, weights TΓ—TT\times T TΓ—TT\times T one query against every key
one head's output TΓ—dvT\times d_v TΓ—64T\times 64 an average of values
MHA(𝐗)\mathrm{MHA}(\mathbf{X}) TΓ—dmodelT\times d_{\text{model}} TΓ—256T\times 256 same shape as 𝐗\mathbf{X}

Return to contents

Slide 14

A small network for each position

Attention mixes information across positions. A two-layer network then works on each position on its own, with the same weights everywhere:

MLP(𝐱⟨t⟩)=𝐖[2] GELU ⁣(𝐖[1]𝐱⟨t⟩+𝐛[1])+𝐛[2]\mathrm{MLP}\big(\mathbf{x}^{\langle t\rangle}\big) = \mathbf{W}^{[2]}\,\mathrm{GELU}\!\big(\mathbf{W}^{[1]}\mathbf{x}^{\langle t\rangle} + \mathbf{b}^{[1]}\big) + \mathbf{b}^{[2]}

Left: the GELU transfer function, Gaussian Error Linear Unit, plotted for inputs from minus 4 to 3, with the ReLU dashed for comparison. GELU is close to 0 for negative inputs, dips slightly below 0 around minus 1, and becomes nearly equal to its input for positive inputs. Right: the widths of the small network in the assignment's model, drawn to scale: a bar for the input x of 256 numbers, an arrow W[1], which maps 256 numbers to 1,024, to a bar four times longer, the hidden layer of 1,024 numbers with GELU applied to each, and an arrow W[2], which maps them back, to a 256-number output

Schematic plus an exact function plot. Left panel, titled GELU: Gaussian Error Linear Unit: output against input z from minus 4 to 3. The ReLU, dashed, is 0 for negative inputs and equal to its input for positive ones. GELU, solid green, follows it closely but smoothly: near 0 for large negative inputs, a small dip to about minus 0.16 at z = minus 1, 0 at z = 0, about 0.84 at z = 1 and 1.95 at z = 2. Right panel, titled the same small network at every position, 4 times wider in the middle: three horizontal bars drawn to scale, the input x of 256 numbers, the hidden layer of 1,024 numbers, four times longer, with GELU applied to each number, and the output of 256 numbers; magenta arrows between them are labelled W[1], 256 to 1,024, and W[2], 1,024 to 256.

Open full-size figure

Return to contents

Slide 15

Residual connections and layer normalization

  • Residual: each sublayer adds its output to its input, 𝐗←𝐗+Sublayer(𝐗)\mathbf{X}\leftarrow\mathbf{X}+\mathrm{Sublayer}(\mathbf{X}), as in ResNets. The stream runs straight through; attention and the MLP only add to it
  • LayerNorm: rescale each position's vector to mean 0 and variance 1, then a learned scale and shift
  • Pre-norm: LayerNorm at the start of each branch, not on the stream
  • One block = both branches. Blocks stack: the model you train has 4
One pre-norm transformer block, read from the bottom up. The residual stream x enters at the bottom and runs straight up the left side to the next block. A first branch leaves the stream, goes through LayerNorm and then multi-head attention, and its output is added back to the stream at a plus sign. A second branch then leaves the stream, goes through LayerNorm and the small network (MLP), and is added back at a second plus sign. LayerNorm sits at the start of each branch, never on the stream itself

Schematic of one pre-norm transformer block, read from the bottom up. The residual stream, a vector x for each position, enters at the bottom from the block below and runs straight up the left side to the next block. A first branch leaves the stream to the right, passes through a LayerNorm box and then a gold box, multi-head attention, and its output returns left to a circled plus sign on the stream, where it is added. Above that, a second branch leaves the stream, passes through a second LayerNorm box and a green box, the small network (MLP), and returns to a second circled plus sign. LayerNorm sits at the start of each branch, never on the stream itself; attention and the MLP each add to the stream rather than replace it.

Open full-size figure

Return to contents

Slide 16

Why copying takes two heads in a row

Two bands of word boxes for The black cat sat on the mat. The black, the last black marked as the current position. Top band, block 1, a previous-token head: an arrow from each word to the next, and the first cat receives a tag reading previous: black. Bottom band, block 2, an induction head: the query of the last black, find previous = black, matches the key of the first cat, previous: black, along a curved gold arrow; the value of cat, cat, flows to a box reading predict: cat. A faint dashed arrow from the last black back to the first black is labelled one head alone: finds black, reads back black

Schematic. Two horizontal bands of word boxes for "The black cat sat on the mat. The black", with the last "black" marked as the current position. Top band, block 1, a previous-token head: a short arrow from each word to the next, so each position receives information about the word just before it; the first "cat" receives a tag reading "previous: black". Bottom band, block 2, an induction head: the query of the last "black", "find previous = black", matches the key of the first "cat", which now carries "previous: black". A curved gold arrow joins them, and the value of that position, "cat", flows to a box reading "predict: cat". A faint dashed arrow from the last "black" back to the first "black" is labelled "one head alone: finds black, reads back black". A single head can find the earlier copy of the current word, but only a head reading the tag left by an earlier head can find the word that came next. That is why copying needs two heads in a row.

Open full-size figure
  1. Problem (dashed): one head finds the earlier black, but reads back black
  2. Block 1: each token notes the token before it; the first cat gets "previous: black"
  3. Block 2: the second black seeks "previous: black", finds cat: predict cat

Return to contents

Slide 17

Try it: inside a transformer

Return to contents

Slide 18

Training a model to write

What do we train it to do, and how do we know it learned anything?

Return to contents

Slide 19

One objective: predict the next character

L=βˆ’1Tβˆ‘t=1Tlog⁑y^ y⟨t+1⟩⟨t⟩\mathcal{L}=-\frac{1}{T}\sum_{t=1}^{T}\log \hat{y}^{\langle t\rangle}_{\,y^{\langle t+1\rangle}}

Two rows of character boxes. The bottom row, in blue, is the input: t, h, e, underscore for a space, c, a, t, underscore, s, a. The top row, in green, is the target at each position: the next character, h, e, underscore, c, a, t, underscore, s, a, t. A gray arrow runs up from each input box to its target

Schematic. Two rows of character boxes. The bottom row, in blue, is the input: t, h, e, an underscore for a space, c, a, t, underscore, s, a. The top row, in green, is the target at each position, the next character: h, e, underscore, c, a, t, underscore, s, a, t. A gray arrow runs up from each input box to its target. One window of text gives as many training examples as it has characters, and the causal mask keeps each position from seeing its own target.

Open full-size figure

Return to contents

Slide 20

Batches and the training loop

Return to contents

Slide 21

Generating text: sample, append, repeat

  1. Feed in a prompt
  2. Read the model's probabilities for the next token, at the last position
  3. Draw one token from them and append it to the prompt
  4. Repeat; keep only the last TT tokens if the text grows longer than the context

Return to contents

Slide 22

Temperature in a real model

Prompt: The black cat sat on the mat. Then Β· GPT-2 small, 40 new word pieces, same seed

Real computation: GPT-2 small, pure sampling, torch seed 0; "β‰ˆ 7": like a choice among 7 equally likely pieces

Return to contents

Slide 23

What did the model learn?

Read its heads, then lesion them

Return to contents

Slide 24

Three ways to open a model

Return to contents

Slide 25

What heads do, and what their weights cannot tell you

Return to contents

Slide 26

A behavior to explain: copying a new name

Return to contents

Slide 27

Knock out a head: a lesion study

Return to contents

Slide 28

The same lesion in GPT-2

Two bar charts of GPT-2 small's loss in nats per token. Left, copying the second half of a repeated random sequence: intact 0.25, the 8 heads with the highest induction weight knocked out 7.49, 8 random other heads knocked out 0.81 with a range bar up to 2.7. Right, ordinary English sentences: intact 4.69, induction heads out 5.25, random heads out 4.83

Computed from GPT-2 small. Two bar charts of loss in nats per word piece, lower meaning better prediction. Left, copying: predicting the second half of a sequence of 50 random word pieces repeated twice, which can only be done by copying. Intact model, 0.25. With the 8 heads of highest induction weight set to zero, 7.49, about thirty times the intact loss and approaching a uniform guess (10.8). With 8 random other heads set to zero, 0.81 on average over 10 draws, with a range bar up to 2.7. Right, ordinary English: 20 sentences. Intact, 4.69; induction heads out, 5.25; random heads out, 4.83. Removing 8 of 144 heads raises the copying loss about thirtyfold and the ordinary-text loss by 0.56 nats.

Open full-size figure

Real computation: GPT-2 small, 144 heads, ranked by induction weight on repeated random tokens; knockouts zero the head's output

Return to contents

Slide 29

The converse test: put the activity back

Three rows of character boxes for a short story, each ending in the second mention Z e f a m. Row 1, Clean (donor): the earlier name is Zefam; loss on efam low. Row 2, Corrupted: the earlier name is Molly; loss on efam high. Row 3, Patched: the corrupted story, with gold arrows bringing one head's output at the four positions Z, e, f, a of the second mention down from the clean row; loss on efam: question mark. Below: earlier letters are the true text, the model is not writing the name freely

Schematic of the assignment's patching experiment. Three rows of character boxes for the same short story, each ending in a second mention of the name Z e f a m. Row 1, Clean (the donor): the earlier mention is also Zefam, so the model can copy, and its loss on the letters e, f, a, m is low. Row 2, Corrupted: the earlier mention is Molly, so there is nothing to copy, and the loss on e, f, a, m is high. Row 3, Patched: the corrupted story, with gold arrows bringing one head's output at the four positions Z, e, f, a (the positions that predict the next letter) down from the clean row; the loss on e, f, a, m is a question mark, the quantity the experiment measures. A note below states that the earlier letters are the true text, so the model is never writing the name freely. If putting one head's clean output back lowers the loss toward the clean value, that head carries the copying signal.

Open full-size figure

Return to contents

Slide 30

What attention gains, and what it costs

Return to contents

Slide 31

Recap

Return to contents

Slide 32

Minute paper

Submit on Canvas β†’ Minute papers β†’ Minute Paper 17 (access code read out in class).

Write three brief points in your own words:

  1. A language model is only trained to predict the next word. How can it end up
    copying a name it has seen once earlier in a story?
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents

Slide 33

Further reading for the research trail

Optional: a few research-trail entries over the semester. Any paper mentioned in any lecture qualifies, but papers from these lectures are preferred.

Return to contents

Slide 34

Going further

Optional: heads in long documents and in vision, sampling in real models, another circuit

Return to contents

Slide 35

Twenty heads find a fact in a long document

Figure 1 of Wu et al.: two grids of needle-in-a-haystack accuracy, context length from 5K to 100K tokens across, depth of the hidden sentence from 0.1 to 0.9 down. Left, with the top 20 retrieval heads masked, many cells turn yellow and red, accuracy 63.6. Right, with 20 random heads masked, almost every cell stays green, accuracy 94.7

Reproduced from Wu et al. (2025), Figure 1, for LLaMA-2-7B-80K. Two grids of the "needle in a haystack" test. In each grid, context length runs across from 5,000 to 100,000 tokens and the depth at which a fact was hidden in the document runs down from 0.1 (near the start) to 0.9 (near the end). Each cell is one test: green is a correct answer to a question about the hidden fact, yellow to red increasingly wrong. Left, with the 20 highest-scoring retrieval heads masked: many cells are yellow and red, and overall accuracy is 63.6%; many of the wrong answers are made up. Right, with 20 random heads masked: almost every cell stays green, accuracy 94.7%. Masking 20 of the model's 1,024 heads lowers accuracy from 94.7% (20 random heads masked) to 63.6%.

Open full-size figure

Return to contents

Slide 36

The same mechanism in vision

Figure 1 of Caron et al.: eight photographs, each followed by the self-attention map of a DINO vision transformer, bright where the class token puts its weight; the bright patches outline the bird, the toothbrush, the building, two giraffes, a boat, a bicycle, a dog and a plane, with no label, mask or box given

Reproduced from Caron et al. (2021), Figure 1. Eight photographs in two rows, each followed by a dark purple map of the same size: a yellow bird on a branch, a toothbrush on a striped cushion, the Houses of Parliament across the Thames, two giraffes, a sailboat, a bicycle against a building, a dog asleep in an armchair, and a small airplane over a runway. Each map is the attention of the class token in the last block of a DINO vision transformer, one weight per 8 by 8 pixel patch, bright where the weight is high; with images several hundred pixels wide this is a grid of dozens of patches on each side (60 by 60 at 480 pixels), which is why the maps look fine-grained. In every case the bright patches trace the object's outline (the bird, the toothbrush, the building's towers, both giraffes, the boat's mast and hull, the bicycle's wheels, the dog, the plane) and the background stays dark. The network was trained without labels, masks or boxes.

Open full-size figure

Return to contents

Slide 37

Where the bright patches come from

Return to contents

Slide 38

Attention sinks in vision, and what they hide

Figure 1 of Jiang et al.: an eagle and a dog, then attention maps from OpenCLIP and DINOv2 in three versions each. Original: a few bright background patches and little else. Shifted: the bright patches rearranged into a smiley face or a heart. With a test-time register: the map outlines the bird and the dog

Reproduced from Jiang et al. (2025), Figure 1. Two photographs, an eagle and a dog, each followed by attention maps from two vision transformers, OpenCLIP and DINOv2, in three versions. Original: the map as the trained model computes it, a few very bright patches scattered over the background and little on the animal. Shifted: the authors move the high activations to patches of their choosing, so the bright spots form a smiley face or a heart, showing that their location is arbitrary. With a test-time register: the high activations are moved to one extra token added to the input, and the remaining map outlines the eagle and the dog. No model is retrained. The bright background patches are the vision counterpart of the first-token sink in GPT-2.

Open full-size figure

Return to contents

Slide 39

The same recipe in real systems

Does the toy recipe survive scale?

Return to contents

Slide 40

Try it: temperature and sampling

Return to contents

Slide 41

The loss ladder with real models

Horizontal bars of held-out loss in nats per character on the same 40 TinyStories stories: uniform guess 4.57, bigram table 2.30, the assignment's 4-block character GPT 0.60, GPT-2 small 0.60, GPT-2 medium 0.53, GPT-2 large 0.49, and TinyStories-33M, trained on the TinyStories training split, 0.34

Computed. Horizontal bars of held-out loss, in nats per character, on the same text, the first 40 stories of the TinyStories validation split; shorter is better. From top to bottom: a uniform guess over 97 characters, 4.57; the bigram table, 2.30; the assignment's 4-block character model, 0.60; GPT-2 small, 0.60; GPT-2 medium, 0.53; GPT-2 large, 0.49; TinyStories-33M, a model trained on the training split of this collection, 0.34. Word-piece models' losses are converted to nats per character by dividing by the characters their pieces cover. A small model trained on this kind of text matches GPT-2 small, trained on 40 GB of web text, and a specialist beats them all.

Open full-size figure

Real computation: the same 36,000 held-out characters scored by every model; word-piece losses converted to nats per character

Return to contents

Slide 42

A different behavior, a different circuit

Figure 1a of Yin and Steinhardt: few-shot in-context-learning accuracy of Pythia 6.9B as a growing percentage of its attention heads is ablated. Clean and random-ablation lines stay flat near 0.56; ablating induction heads lowers accuracy slowly to 0.44 at 20 percent; ablating function-vector heads drops it to about 0.32 at 3 percent, 0.20 at 9 percent and 0.18 at 15 percent

Reproduced from Yin and Steinhardt (2025), Figure 1a, for Pythia 6.9B. A line plot of few-shot in-context-learning accuracy (vertical, 0.2 to 0.55) against the percentage of attention heads ablated (horizontal, 1 to 20%). Four lines. Clean, yellow, is flat at about 0.56. Random ablation, green, stays near 0.55, ending at 0.54. Ablating induction heads, blue, lowers accuracy slowly, from 0.55 at 1% to 0.44 at 20%. Ablating function-vector (FV) heads, pink, drops it steeply: 0.46 at 1%, about 0.32 at 3%, 0.20 at 9% and 0.18 at 15 and 20%. For learning a task from examples, as opposed to copying, function-vector heads matter far more than induction heads.

Open full-size figure

In the figure: ICL = in-context learning, a task learned from examples in the prompt Β· FV = function-vector heads Β· ablation = knockout

Return to contents