Three tokens, here whole words: the cat sat. Update the last position, sat (). Its query is ; keys and values are made up, two numbers each, so :
| the | cat | sat | |
|---|---|---|---|
| key | |||
| score | |||
| score | |||
| weight , softmax | |||
| value |
, mostly cat's value

Schematic: hand-set scores, softmax computed · every row sums to 1
Schematic: hand-set scores, softmax computed · every row sums to 1
| Model | Context it uses | Held-out loss (nats) | Effective choices, |
|---|---|---|---|
| Uniform guess | none | 4.57 | 97 |
| Bigram table | the current character | 2.28 | 9.8 |
| Average | every earlier character, equal weights | 2.03 | 7.6 |
| One attention head | every earlier character, chosen weights | 1.23 | 3.4 |
Optional: a few research-trail entries over the semester. Any paper mentioned in any lecture qualifies, but papers from these lectures are preferred.
Submit on Canvas → Minute papers → Minute Paper 16 (access code read out in class).
Write three brief points in your own words:
Credit for a thoughtful attempt, not for being correct.
- This lecture opens Theme 3, on sequences: text, speech, a stream of experience. Until now every input was taken in one piece (an image, a set of ratings). From now on the input is a sequence, and the question is how a model uses what came before - Until about 2017, each field of AI had its own kind of network, designed for its kind of data: convolutional networks for images, recurrent networks for text and speech, sequence-to-sequence models for translation - You met convolutional networks in Theme 2; recurrent networks come back in a later lecture
- Within a few years of 2017, the transformer replaced the specialized networks in almost every field. Vaswani et al. (2017) introduced it for translation; the citation count is Semantic Scholar's, November 2026 - Each system arranges and trains its blocks in its own way and adds parts of its own (AlphaFold adds a good deal). What they share is the transformer block and, in it, attention - Systems in the tiles: DINOv2 (Oquab et al. 2023) and Segment Anything (Kirillov et al. 2023) for images; Stable Diffusion 3 (Esser et al., ICML 2024) and Sora (OpenAI 2024) generate images and video from patches; Whisper (Radford et al., ICML 2023) transcribes speech; Decision Transformer (Chen et al. 2021) and RT-2 (Brohan et al. 2023) treat an agent's or a robot's actions as tokens; AlphaFold 3 (Abramson et al. 2024) predicts the structure of proteins and the molecules they bind
- Cutting the input into tokens is called tokenization. Text is cut into characters or pieces of words (common words are one token, rare words a few); an image into a grid of patches, often 16 × 16 pixels; a sound into short slices of its spectrogram, about 20 ms each; a protein into its amino acids - After this step the transformer sees only a sequence of vectors, one per token. A model can even mix them: text tokens and image tokens in one sequence - The waveform in the figure is made up for the drawing; the tiger is the photograph from the first lecture
- The lines of the attention band only go back in time: a token never reads a later one, because the model's job is to predict that later token. This is causal self-attention; the causal mask that enforces it comes later today - Read the figure from the bottom up, one column per token. The only place where one token learns anything about another is self-attention; everything else works on each column separately. That is why a transformer without attention would be a bag of unrelated tokens - The tokens here are characters, as in today's running example and the assignment. Real language models cut text into pieces of words; the next lecture shows how those are chosen. The machinery is the same - The gray line running up each column carries the token's vector from block to block; each step adds its result to it. It is called the residual stream, and the next lecture builds it - The small network has the same weights at every position within a block; each block has its own - Each step starts with a layer normalization, left out of the drawing for clarity. The next lecture assembles a complete block, normalizations included, and Part 2 of the assignment builds and trains one - A block is also called a layer, the word most papers use. GPT-2 small has 12 blocks of 12 heads each with vectors of 768 numbers; GPT-3 has 96 blocks. The design of the block is the same; only the number of blocks, the width and the training data grow
- First the old pieces. Every one of them was in use before 2017
- Bengio et al. (2003), "A neural probabilistic language model": each word gets a learned vector, its embedding, and a small network predicts the next word from the vectors of the few words before it. That is the first step of every transformer today, and the same next-word task as this lecture. Word2vec (Mikolov et al. 2013) later made word embeddings widely used - Attention itself is older than the transformer: Bahdanau et al. (2015) let a translation network look back at the words of the input sentence, and a few 2016–2017 models applied attention of a sequence on itself inside recurrent networks. The transformer's step was to build the whole network from self-attention, without recurrence - The next lecture assembles the full block: two normalizations, attention, a small network, and two residual additions
- The figure is from the first transformer paper: two of the eight heads in layer 5 (of 6) of the part of the model that reads the English sentence. The word on the right (its) is the one reading; the words on the left are the ones it reads. The thicker and darker the line and the bar, the more weight that word gets - Linking its to Law is called anaphora resolution: an anaphor is a word, such as a pronoun, that points back to something mentioned earlier, and resolving it means finding what it points to - This part of the model sees the whole sentence, so its may also read application, which comes after it. The models in the rest of this lecture predict the next word, so they may only read words that come before - One sentence shows what two heads do on that sentence, not what they do in general; the authors call them "apparently involved in anaphora resolution". The next lecture measures heads of GPT-2 over many sentences - The paper prints the words sideways; here the panel is rotated so that they read across, and cut after "should be just"
- The next three slides are quick tests on yourself: readers guess what comes next and see what they expect. Then the same job, predicting the next character, is given to a model, and the rest of this part asks how much of the text before it the model should use: one character, all of it, or the parts that matter. That last step is attention
- Read the triangle once, quickly, before going on
- Answer quickly, before going on to the next slide
- The doubled "the" sits at a line break, where the eye expects a new word; readers see the phrase they predict - The Moses illusion (Erickson & Mattson 1981): Moses fits the question well enough (a biblical figure, like Noah), so the mismatch goes unnoticed. People who know the story still answer "two" - Rayner, White, Johnson and Liversedge (2006) tracked readers' eyes: text with letters swapped inside words is still read, but more slowly, and swaps at the start of a word cost the most. Prediction helps, but readers still use the letters - Kutas and Hillyard (1980) recorded EEG while people read sentences one word at a time. A word that makes no sense where it appears, such as socks after "spread the warm bread with", evokes a negative brain wave about 400 ms after it appears, the N400; the more expected the word, the smaller the wave - A later lecture measures this directly: words that are hard to predict take longer to read
- TinyStories (Eldan & Li 2023) is a set of short stories written by GPT-3.5 and GPT-4, using only words a three- or four-year-old would know. The authors built it to ask how small a language model can be and still write fluent, consistent English; it has since become a standard training set for small models, including those of Assignment 4 - Characters keep the examples small: 97 symbols, every one visible on a slide. Real models predict pieces of words; the next lecture shows that the same machinery works on them
- Recurrent networks dominated language modelling before transformers. Their single memory vector is a bottleneck: a name read long ago must survive many updates to be used. A later lecture covers them and what they still offer - A transformer has no such bottleneck: any earlier position can be read directly, in one step. When this lecture says a position reads "the past", it means the positions to its left in the window, all of which the model has at once - Training uses this: every position of the window predicts its own next token at the same time
- Text drawn at random from a character bigram table of English has English-looking letter pairs but few real words; longer contexts give more real words - Shannon (1948) built tables like this from English text, for pairs of letters and longer sequences, and showed that text generated from them looks more like English as the context grows. Counting tables of words (n-grams) were the standard language models of speech recognition and machine translation until neural networks replaced them in the 2010s
- Adding 1 to every count is called add-one (Laplace) smoothing. It costs little: each row sum grows by the number of symbols, here 5
- In the toy table, an underscore stands for a space. All three observed transitions from a lead to t (the a at the very end has no successor), so before smoothing P(t | a) = 3/3 = 1, and after smoothing (3 + 1)/(3 + 5) = 1/2; two of the six transitions from t lead to e, so P(e | t) goes from 2/6 to 3/11 - The loss is easier to read as e^loss, the effective number of choices: the number of equally likely next characters that would give the same loss. A uniform guess over 97 characters gives 97; a perfect model gives 1. The assignment's report prints it - Before scoring held-out text, add 1 to every count (add-one smoothing): a pair never seen in training would otherwise get probability 0, and an infinite loss. In the toy table, smoothing turns P(t | a) = 3/3 into (3 + 1)/(3 + 5) = 4/8, with five symbols - Shannon (1948) generated text from letter-pair ("digram") and longer letter statistics of English, and showed that the text looks more like English as the context grows - In the lecture on baselines, a score meant something only next to a baseline computed on the same data. The same rule applies here: compare on the same held-out text
- A sure and correct guess (p = 1) costs 0; a guess among 97 equally likely characters costs ln 97 = 4.6 - Averaging −ln p is the same as taking minus the log of the product of all the p's, divided by the number of positions: the cross-entropy measures how probable the model finds the whole held-out text, per character - This is the same loss as for class probabilities in the backpropagation lecture: here the classes are the 97 possible next characters - e to the power of the loss is easier to read: the effective number of choices. A uniform guess over 97 characters gives 97; a perfect model gives 1. Here e^1.34 = 3.8: on these two positions the table is as uncertain as a choice among about four characters - The two numbers come from the smoothed table of the last slide: P(e | t) = 3/11 and P(a | e) = 2/8 = 1/4 - Held-out text matters: on its own training text, a model can score well by memorizing. The lecture on baselines made the same point: compare models on the same held-out text
- With four characters of context there are about 89 million possible contexts, so a table of about 8.6 billion counts; a training text of 250 million characters cannot fill it - The table also treats "the cat" and "the dog" as unrelated contexts. With vectors, a model can give related symbols similar vectors, so what it learns about one context can carry over to similar ones
- An example with word tokens: cat and dog appear in the same places (the ___ sat, feed the ___), so training gives them similar vectors, and what the model learns about "the cat sat" carries over to "the dog sat". King and queen are another such pair - Why not keep the one-hot codes? They make every pair of characters equally far apart, so nothing learned about one character carries over to another. Learned vectors can put characters that behave alike, such as the vowels, close together. That matters next: attention scores positions with dot products, and between one-hot vectors of different tokens every dot product is 0 - The embedding table is a weight matrix of size (vocabulary size × d_model), learned by gradient descent like any other weights. Looking up row k gives exactly what multiplying the one-hot vector of token k by the table would give, without doing the multiplication - Size: the model of Assignment 4, Part 1 gives each of its 97 characters 128 numbers, more than the one-hot code, because the point is similarity, not saving space. For words the table also saves space: GPT-2 maps about 50,000 tokens to 768 numbers each - T is the length of the window of text the model reads at once, the number of tokens; d_model is the width of every vector inside the model. Each position's vector is a row of X, and subscripts i and j name positions; this matches the papers and the (batch, sequence, width) layout of tensors in PyTorch - Here every token is a character. Real language models use pieces of words as tokens; the next lecture covers how they are chosen. The mechanism is the same: one row per token
- e_i is the row of the embedding table for the token at position i, and p_i the row of the position table for position i; both tables are learned. The step-by-step demo later in the lecture writes the same thing as X = E + P - The vectors x_i now carry both what the token is and where it sits, and every later step can use either - Without position embeddings, the averaging and attention steps would give the same set of outputs for any shuffle of the input tokens; only the causal mask, later, adds a sense of order
- Each position averages over itself and the positions before it, never the ones after: when predicting, the future is not available. A model that uses only the past like this is called causal - The vectors are made up, two numbers each, so the arithmetic fits on a slide; a trained embedding table learns its own. Three words keep the example readable; characters work the same way - Here y_3 happens to equal sat's own vector, a coincidence of the made-up numbers - As a matrix product: multiply the stacked vectors by a fixed T × T averaging matrix whose row t holds 1/t in its first t entries and 0 after. The assignment builds the average this way
- The output layer is a linear readout, like the perceptron's, with the same weights at every position - Training adjusts the embedding table, the output layer and, later, the matrices of attention, all together, to lower the cross-entropy of the next character
- With all scores equal, every weight is 1/i and attention is the average of two slides back - This softmax is the same function as the output layer's, used differently: there it chooses among 97 characters; here it spreads position i's weight over the positions 1 to i - Scoring with the raw vectors x is the simplest version. The next slides give each position separate vectors for asking (the query), for being matched (the key) and for being read (the value), each made by a learned matrix; nobody gives the model target weights
- This is the attention step alone, without a transformer around it and without learning: the keys, values and query are written by hand, and each description is a short vector of features (red, blue, triangle, circle) - In a transformer, every token plays all three roles at once: it asks with a query, is matched through its key and passes on its value. The next slides build that from the token vectors
- The story: three objects were put away. The red triangle is on the desk, the red circle on the shelf, the blue triangle in the drawer. Each object's description is its key; where it is, its value - The question "where is the red triangle?" is the query. Attention compares the query with every key, turns the matches into weights, and returns the weighted mix of the values: the bars show how much of each location comes back - 1. Case 1, "A useful cue", opens first. Before pressing anything, guess which location comes back most. Press "Reveal computation": desk 90.9%, shelf 4.5%, drawer 4.5%. The red triangle's key matches both "red" and "triangle"; the others match only one of the two - 2. On the right, the slider "Scores multiplied by" sets how strongly the best match wins (6 at the start). Drag it to 0: every object gets 33.3%, the plain average. Drag it back up: desk takes over again. The multiplier is an inverse temperature: a large one makes the softmax pick the best match, 0 makes it the plain average. Dividing by √d_k, later today, is a fixed version of the same knob - 3. Case 2, "An incomplete cue": the question now says only "red". Reveal: desk and shelf tie at 48.8%, since both are red. Drag the slider to 12: still a tie (49.9% each). Tick "Include shape in the query": desk is back to 90.9% - 4. Case 3, "Many distractors": one red triangle among 24 red circles. Reveal: desk gets 45.6%, while the circles, each a weak match, add up to 54.4%. Move "Similar distractors" down to 5 (desk 80.1%), or the slider up to 12 (desk 94.4%). The reason is the softmax's normalization: all weights share a total of 1, so 24 weak matches together can outweigh one strong match. Visual search shows the same set-size effect: a target that shares features with many distractors becomes harder to find as the display grows - 5. Case 4, "No correct source": the red triangle's entry is missing. Reveal: shelf gets 95.3%, a confident answer that is wrong. Tick "Restore the matching source": desk 95.0% - The point: attention can only read what is in its context, and a high weight on one source does not mean the answer is right. "Show the math" lists every score and weight - The objects and locations are made up for the demo; they are not words, a trained model, or data from human memory
- Position i is the i-th token of the window. The same three matrices are used at every position - Separate vectors let a token be found for one reason and pass on something else: a name can be matched on "a name was here" and pass on its letters - d_k is the number of entries in each query and key; dividing by √d_k keeps the scores in a range where the softmax works well (a later slide explains why) - Think of a library catalog: the query is your search, the keys are the index cards, the values are the books. Unlike a catalog, which hands you one book, attention returns a mix of all the values, weighted by the matches
- Each projection is a linear layer without the nonlinearity, applied to every row of X with the same weights. W_Q and W_K have size d_model × d_k, W_V has size d_model × d_v - W_Q, W_K and W_V are fixed once training ends; the queries, keys and values are recomputed for every new input - d_k is the width of queries and keys (they must match, because we take their dot product); d_v is the width of values
- Row i of A holds position i's weights a_ij: nonnegative, summing to 1. Row i of A V is y_i, the new vector of position i - Why √d_k: when the entries of a query and a key are roughly independent, with mean 0 and variance 1, their dot product has variance d_k, a spread of √d_k. Without the division, large scores push the softmax to put almost all weight on one position, and its gradient becomes close to zero - The softmax is the one from the lecture on learning the weights, where it turned a network's scores into class probabilities. Here it turns one row of scores into weights that sum to 1
- In a trained model the query, keys and values come from the learned matrices W_Q, W_K and W_V; here they are made up so the arithmetic fits on a slide - Scores: (1,0)·(0,1) = 0; (1,0)·(2,0) = 2; (1,0)·(1,1) = 1. Divided by √2: 0, 1.414, 0.707 - Softmax: e^0 = 1, e^1.414 = 4.11, e^0.707 = 2.03; their sum is 7.14, so the weights are 0.14, 0.58 and 0.28, which add up to 1 - The key of cat matches the query of sat best, so cat gets the largest weight, and y_3 is pulled toward cat's value. Every other position does the same with its own query, at the same time
- The same three words and numbers as "Attention by hand". The demo updates sat, the last word, unless you click another word at the top. Press Next to go one stage at a time - 1. Input: each word's vector is its word embedding plus its position embedding; for sat, x3 = [1, 1, 0] - 2. Q, K, V: three matrices, W_Q, W_K and W_V, turn each vector into a query, a key and a value. The same three matrices are used for every word - 3. Match: sat's query is compared with every key by a dot product, giving the scores 0, 2 and 1 - 4. Scale and mask: each score is divided by √2, because queries and keys here have 2 numbers; dividing by the square root of their length keeps the scores from growing with the vector length. The mask hides words that come after the one being updated - 5. Softmax: the scores become the weights 0.14, 0.58 and 0.28, which add up to 1 - 6. Gather: the new vector for sat is the weighted sum of the values, (0.28, 0.72) - 7. All positions: every word does the same at the same time - To see the causal mask at work, first click "the" at the top: the is the first word, so with the mask it can read only itself (weight 1). Untick "Causal mask": now the reads cat and sat too. For sat the mask changes nothing, because no word comes after it - Other things to try, one at a time: untick "Divide by √2" (the weights become more uneven); tick "Equal weights" (every allowed word gets the same weight, the plain average); tick "Zero cat's value" (cat still gets the largest weight, but brings nothing back)
- The same three words as "Attention by hand". The magenta arrow is the query of the word chosen under "asking" (sat at the start); the blue arrows are the keys of the, cat and sat. A thicker key arrow means more weight; a gray key is a later word, hidden by the causal mask - 1. The demo opens on the numbers of "Attention by hand": weights 0.14, 0.58 and 0.28, output (0.28, 0.72). The lower plot shows the three values (green) and the output, their weighted average (magenta) - 2. Drag the query toward cat's key: cat's weight grows, and the output moves toward cat's value - 3. Move the query up and to the left, to about (−2, 2): the scores become 2, −4 and 0, so the gets the largest weight - 4. Keep the direction and double the length: the scores double and the weights become more uneven. Unticking "divide by √d_k" does the same - 5. Click cat under "asking": the query becomes cat's own, (1, 1), and sat's key turns gray, because sat comes after cat (its gray arrow sits under cat's query, which happens to be the same vector). Weights 0.33 and 0.67 on the and cat. Untick "causal mask": sat can now be read too
- Read one row at a time. A row is one query word; the colors across it are its weights over the six key words, and they add up to 1 - The scores here were set by hand to make the pattern easy to see. In a trained model they come from W_Q and W_K, which are learned - The right panel is the causal mask, the next slide
- The mask matters because training is parallel: the whole sequence goes in at once, and position t is asked to predict the symbol at t + 1. The mask keeps each position's prediction to the symbols before it, as at generation time - Setting a score to minus infinity gives e^(−∞) = 0 in the softmax, and each row renormalizes over the positions to its left. The first row can only read itself - The diagonal is allowed: every position reads itself and the positions before it
- These are the numbers that Part 1 of the assignment prints for each model, on the same held-out stories. The effective number of choices is e to the power of the loss: the attention head is as uncertain as a choice among about 3.4 equally likely characters, against 97 for a uniform guess - The bigram table is the baseline from the start of the lecture: a model that uses more context is worth something only if it beats it on the same held-out text
- Next lecture: what trained heads do in real models such as GPT-2, then stack attention into a full model, train it to write stories, and look inside it to see what its heads do
- Adeli et al. (2026) and Mehrani & Tsotsos (2023) make a pair: the first shows that DINO-style transformers predict human reaction times in a same-object task from their patch affinities; the second argues that what these models compute is grouping by similarity, not attention in the psychologist's sense. Decide for yourself whether the word is earned - Christian et al. (2026) compare six vision–language models' attention maps with human fixations on the same images; the match depends mostly on the kind of decoder - Barbero et al. (2025) explain why so many heads park their weight on the first token, which the next lecture shows in GPT-2 and in vision transformers: parking it there keeps a deep model from over-mixing its positions - Goldstein et al. (2025) and Cai et al. (2026) are where the model–brain program stands for language: a speech model's layers aligned with the cortical hierarchy during real conversation, and single frontal neurons sorted by what a language model's representations say about each word - Two more that qualify: Lindsay (Front Comput Neurosci 2020, doi:10.3389/fncom.2020.00029), a review of attention across psychology, neuroscience and machine learning; Bricken & Pehlevan (NeurIPS 2021, arXiv:2111.05498), attention as a cerebellar associative memory, a bridge to the memory models of a later lecture - Before posting, skim the existing entries on your paper; your post must add something not already said
- Submitted on Canvas (Minute papers → Minute Paper 16); credit for a thoughtful attempt