Attention

Reader-friendly version of the 39 lecture slides: the lecture content in normal flow, with figure descriptions and data tables where the figure carries data. Open the presentation.

Slide 1

Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 15 · Theme 3: Dynamics, sequences & world models

Attention

Tuesday, November 10 · Fall 2026

Return to contents

Slide 2

Before 2017: one architecture per field

Six tiles, one per field of AI before 2017, each with its own kind of network and a small drawing of it. Vision: convolutional networks, drawn as shrinking stacks of feature maps (AlexNet 2012, ResNet 2016). Language: recurrent networks, a chain of cells reading the, cat, sat, on one word at a time. Speech: deep belief nets with hidden Markov models, drawn as stacked layers. Translation: sequence-to-sequence networks, an encoder chain feeding a decoder chain. Decisions and robots: reinforcement learning, an agent and the world in a loop (DQN 2015). Proteins: hand-built pipelines of align, score and fold steps

Six tiles, one per field of AI before 2017, each with its own kind of network and a small drawing of it. Vision: convolutional networks, drawn as shrinking stacks of feature maps (AlexNet 2012, ResNet 2016). Language: recurrent networks, a chain of cells reading the, cat, sat, on one word at a time. Speech: deep belief nets with hidden Markov models, drawn as stacked layers. Translation: sequence-to-sequence networks, an encoder chain feeding a decoder chain. Decisions and robots: reinforcement learning, an agent and the world in a loop (DQN 2015). Proteins: hand-built pipelines of align, score and fold steps

Open full-size figure
Idea after Lucas Beyer's transformer lecture

Return to contents

Slide 3

Today: one architecture wins them all

The same six tiles, now each with the same drawing: a transformer block, a small network above a highlighted attention step, repeated L times. Vision: DINOv2 and Segment Anything; images and video: Stable Diffusion 3 and Sora. Language: ChatGPT, Claude, Gemini. Speech: Whisper, speech recognition in about 100 languages. Translation: the first transformer, 2017, now done by chatbots too. Decisions and robots: Decision Transformer and RT-2, a robot's actions as tokens. Proteins: AlphaFold 2 and 3, structure from the amino-acid sequence

The same six tiles, now each with the same drawing: a transformer block, a small network above a highlighted attention step, repeated L times. Vision: DINOv2 and Segment Anything; images and video: Stable Diffusion 3 and Sora. Language: ChatGPT, Claude, Gemini. Speech: Whisper, speech recognition in about 100 languages. Translation: the first transformer, 2017, now done by chatbots too. Decisions and robots: Decision Transformer and RT-2, a robot's actions as tokens. Proteins: AlphaFold 2 and 3, structure from the amino-acid sequence

Open full-size figure

Return to contents

Slide 4

Step one: everything becomes tokens

Three rows, each read from left to right: input, tokens, one vector per token. Text: the cat sat on the mat is cut into word pieces, the first four shown: the, cat, sat, on. Image: a photograph of a tiger is cut into a 4 by 4 grid of patches, and four patches are shown as tokens. Sound: a waveform is cut into short slices, four of them shown as tokens. In every row, each token becomes a small vector drawn the same way

Three rows, each read from left to right: input, tokens, one vector per token. Text: the cat sat on the mat is cut into word pieces, the first four shown: the, cat, sat, on. Image: a photograph of a tiger is cut into a 4 by 4 grid of patches, and four patches are shown as tokens. Sound: a waveform is cut into short slices, four of them shown as tokens. In every row, each token becomes a small vector drawn the same way

Open full-size figure

Return to contents

Slide 5

Inside a transformer

Schematic of a transformer, read from the bottom up, with one column per token. At the bottom, six tokens, the characters t, h, e, a space drawn as an underscore, c and a. Each becomes a small blue vector, labelled embedding plus position. A grey line runs up from each vector, the token's residual stream. Inside a dashed frame marked times L blocks, a gold self-attention band spans all six columns: lines from every column fan in to the last one, so the last token reads itself and the five earlier tokens and none after it. Above it, each column has its own small box labelled MLP, the small network that works on each token alone. At the top, an output layer and softmax spans the columns and gives each column's predicted next token: h, e, space, c, a, t, the input shifted by one

Schematic of a transformer, read from the bottom up, with one column per token. At the bottom, six tokens, the characters t, h, e, a space drawn as an underscore, c and a. Each becomes a small blue vector, labelled embedding plus position. A grey line runs up from each vector, the token's residual stream. Inside a dashed frame marked times L blocks, a gold self-attention band spans all six columns: lines from every column fan in to the last one, so the last token reads itself and the five earlier tokens and none after it. Above it, each column has its own small box labelled MLP, the small network that works on each token alone. At the top, an output layer and softmax spans the columns and gives each column's predicted next token: h, e, space, c, a, t, the input shifted by one

Open full-size figure
  • Here each token is one character, as in the model you train. Read the figure from the bottom: one column per token
  • Each token becomes a vector: embedding + position embedding
  • LL blocks, each with its own weights, in two steps:
    • Causal self-attention: each token reads only itself and earlier tokens
    • A small network, the same for every token, works on each token alone
  • The output layer predicts the next token

Return to contents

Slide 7

Most of a transformer is not new

Most of a transformer is not new — table
Part of a transformer Where it comes from
Embeddings: each token becomes a learned vector Language models, 2003 old
A small network at every position The multilayer perceptron old
Residual connections ResNets, 2016 old
Layer normalization Ba, Kiros & Hinton, 2016 old
Self-attention: each token reads the earlier ones The transformer, 2017 new

Return to contents

Slide 8

Self-attention at work

Two columns of the same sentence, The Law will never be perfect, but its application should be just. A thick purple line runs from its in the right column to Law in the left column, which is shaded strongly; a thinner pink line runs to application, shaded more lightly. No other word receives a line.

Two columns of the same sentence, The Law will never be perfect, but its application should be just. A thick purple line runs from its in the right column to Law in the left column, which is shaded strongly; a thinner pink line runs to application, shaded more lightly. No other word receives a line.

Open full-size figure

Published: both columns are the same sentence; its (right) reads the words on the left

  • The first transformer translated English into German. Both columns are the English sentence it reads first; its tokens are words, not characters
  • An attention head is one way of weighting the other words. A model runs several heads side by side: this one has 8 in each of its 6 reading blocks. Each color here is one head
  • Both heads put most of their weight on Law, the noun its refers to; one also reads application
  • Nobody told the model about grammar: it learned where to look from 4.5 million English–German sentence pairs

Return to contents

Slide 9

Why attention?

Readers predict what comes next. A model that predicts needs the right part of what came before

Return to contents

Slide 10

Read this

PARIS
IN THE
THE SPRING

Return to contents

Slide 11

Answer this

How many animals of each kind
did Moses take on the ark?

Return to contents

Slide 12

You read what you expect

Return to contents

Slide 13

Predict what comes next

Two rows of character boxes. The bottom row, in blue, is the input: t, h, e, underscore for a space, c, a, t, underscore, s, a. The top row, in green, is the target at each position: the next character, h, e, underscore, c, a, t, underscore, s, a, t. A gray arrow runs up from each input box to its target

Two rows of character boxes. The bottom row, in blue, is the input: t, h, e, underscore for a space, c, a, t, underscore, s, a. The top row, in green, is the target at each position: the next character, h, e, underscore, c, a, t, underscore, s, a, t. A gray arrow runs up from each input box to its target

Open full-size figure

Return to contents

Slide 14

One token at a time, or the whole window at once

One token at a time

  • People read one word after another
  • So do recurrent networks, a later lecture: one memory vector is carried forward, and everything read so far must fit in it. For them tt is a time step

The whole window at once

  • A transformer gets the whole window of text at once: its context
  • For a transformer, tt is a position in the window. Position tt can read every earlier position directly. A causal mask, later today, hides the later ones: they hold what it must predict
  • "Earlier" means earlier in the window: all of the context is there at the same time

Return to contents

Slide 17

One character of context: a bigram table

A five-by-five table for the made-up toy text that hat ate the tea, rows the current symbol and columns the next, for a, e, h, t and the space, drawn as an underscore, with a row-sum column: the smoothed counts divided by their row sum, so every row holds probabilities that add up to 1: for example P(t | a) = 1/2 and P(e | t) = 3/11

A five-by-five table for the made-up toy text that hat ate the tea, rows the current symbol and columns the next, for a, e, h, t and the space, drawn as an underscore, with a row-sum column: the smoothed counts divided by their row sum, so every row holds probabilities that add up to 1: for example P(t | a) = 1/2 and P(e | t) = 3/11

Open full-size figure

A made-up toy text with five symbols: "that hat ate the tea"

  • Long before transformers, the simplest model used one character of context: count how often each symbol follows each other one in the training text
  • One character already says a lot: after q comes u; after a space, a new word starts
  • Such tables were the standard language models for decades, in speech recognition and translation
  • A pair never seen in training gets probability 0, and an infinite loss on new text. Smoothing: add 1 to every count
  • Divide each row by its sum: each row now holds probabilities

Return to contents

Slide 18

Scoring a model: the cross-entropy

Scoring a model: the cross-entropy — table
Toy text tea Next pp from the table −ln⁡p-\ln p
after t e 3/11 1.30
after e a 1/4 1.39
average 1.34

Return to contents

Slide 19

Why not count longer contexts?

Why not count longer contexts? — table
Characters of context Possible contexts
1 97
2 9,409
3 912,673
4 88,529,281

Return to contents

Slide 20

Symbols become vectors

Return to contents

Slide 21

Position embeddings: where is each token?

𝐱i=𝐞i+𝐩i\mathbf{x}_i=\mathbf{e}_i+\mathbf{p}_i

Return to contents

Slide 22

Look back at everything: average

Position ii is the ii-th token of the window, with vector 𝐱i\mathbf{x}_i. The simplest way to use the whole context: give position ii a new vector 𝐲i\mathbf{y}_i, the average of positions 1 to ii

𝐲i=1i∑j≤i𝐱j\mathbf{y}_i = \frac{1}{i}\sum_{j\le i} \mathbf{x}_j

Look back at everything: average — table
the cat sat
vector 𝐱j\mathbf{x}_j (made up) (1,0)(1,0) (0,1)(0,1) (0.5,0.5)(0.5,0.5)
weight (equal) 13\tfrac13 13\tfrac13 13\tfrac13

Return to contents

Slide 23

From the new vector to a prediction

Schematic of the model so far, read from the bottom up, with one column per token: the characters t, h, e, a space drawn as an underscore, c and a. Each becomes a small blue vector x, embedding plus position. A gray line runs up each column. In a taupe band labelled mixing step y, lines of equal width run from every column to the last one: the average, equal weights; next, attention. Above it a bar labelled output layer W out plus softmax spans the columns. Above the last column, small green bars give the probabilities of the next character: t is the tallest, then n, r and the space

Schematic of the model so far, read from the bottom up, with one column per token: the characters t, h, e, a space drawn as an underscore, c and a. Each becomes a small blue vector x, embedding plus position. A gray line runs up each column. In a taupe band labelled mixing step y, lines of equal width run from every column to the last one: the average, equal weights; next, attention. Above it a bar labelled output layer W out plus softmax spans the columns. Above the last column, small green bars give the probabilities of the next character: t is the tallest, then n, r and the space

Open full-size figure
  • At every position ii, the output layer, a weight matrix 𝐖out\mathbf{W}_{\text{out}}, turns 𝐲i\mathbf{y}_i into 97 scores, one per possible next character
  • A softmax turns the 97 scores into probabilities; the cross-entropy trains every weight, as in the backpropagation lecture
  • Only the mixing step looks at other positions. The average was one mixing step; attention, next, is a better one

Return to contents

Slide 24

Choose what matters: weights from similarity

aij=esij∑j′≤iesij′a_{ij}=\frac{e^{s_{ij}}}{\sum_{j'\le i} e^{s_{ij'}}}

Return to contents

Slide 25

Self-attention, step by step

How does one word decide which other words to read?

Return to contents

Slide 26

A memory you can search

A memory you can search — table
key value
red triangle desk
red circle shelf
blue triangle drawer

Return to contents

Slide 27

Try it: can attention find it?

Return to contents

Slide 28

Each position plays three roles

Each position makes three vectors from its own 𝐱\mathbf{x}, each with its own learned matrix:

As in "Choose what matters", the new vector at position ii is a weighted sum with softmax weights; now it sums the values, and the weights come from how well the query matches each key:

𝐲i=∑j≤iaij 𝐯j,aij=e 𝐪i⋅𝐤j/dk∑j′≤ie 𝐪i⋅𝐤j′/dk\mathbf{y}_i=\sum_{j\le i} a_{ij}\,\textcolor{#3F7A6B}{\mathbf{v}_j},\qquad a_{ij}=\frac{e^{\,\textcolor{#B5396B}{\mathbf{q}_i}\cdot\textcolor{#3E6D8E}{\mathbf{k}_j}/\sqrt{d_k}}}{\sum_{j'\le i} e^{\,\textcolor{#B5396B}{\mathbf{q}_i}\cdot\textcolor{#3E6D8E}{\mathbf{k}_{j'}}/\sqrt{d_k}}}

dkd_k: the number of entries in each query and key (the next slides explain the division)

Return to contents

Slide 29

Queries, keys and values come from the same input

𝐐=𝐗𝐖Q,𝐊=𝐗𝐖K,𝐕=𝐗𝐖V\textcolor{#B5396B}{\mathbf{Q}}=\mathbf{X}\mathbf{W}_Q,\qquad \textcolor{#3E6D8E}{\mathbf{K}}=\mathbf{X}\mathbf{W}_K,\qquad \textcolor{#3F7A6B}{\mathbf{V}}=\mathbf{X}\mathbf{W}_V

Return to contents

Slide 30

Scaled dot-product attention

Attention(𝐐,𝐊,𝐕)=softmax ⁣(𝐐𝐊⊤dk)⏟𝐀 𝐕\mathrm{Attention}(\textcolor{#B5396B}{\mathbf{Q}},\textcolor{#3E6D8E}{\mathbf{K}},\textcolor{#3F7A6B}{\mathbf{V}})=\underbrace{\mathrm{softmax}\!\left(\frac{\textcolor{#B5396B}{\mathbf{Q}}\textcolor{#3E6D8E}{\mathbf{K}}^\top}{\sqrt{d_k}}\right)}_{\mathbf{A}}\,\textcolor{#3F7A6B}{\mathbf{V}}

Return to contents

Slide 31

Attention by hand: three words

Three tokens, here whole words: the cat sat. Update the last position, sat (i=3i = 3). Its query is 𝐪3=(1,0)\textcolor{#B5396B}{\mathbf{q}_3}=(1,0); keys and values are made up, two numbers each, so dk=2d_k = 2:

Attention by hand: three words — table
the cat sat
key 𝐤j\textcolor{#3E6D8E}{\mathbf{k}_j} (0,1)(0,1) (2,0)(2,0) (1,1)(1,1)
score 𝐪3⋅𝐤j\textcolor{#B5396B}{\mathbf{q}_3}\cdot\textcolor{#3E6D8E}{\mathbf{k}_j} 00 22 11
score ÷2\div\sqrt{2} 00 1.411.41 0.710.71
weight a3ja_{3j}, softmax 0.140.14 0.580.58 0.280.28
value 𝐯j\textcolor{#3F7A6B}{\mathbf{v}_j} (1,0)(1,0) (0,1)(0,1) (0.5,0.5)(0.5,0.5)

𝐲3=0.14 (1,0)+0.58 (0,1)+0.28 (0.5,0.5)=(0.28, 0.72)\mathbf{y}_3 = 0.14\,(1,0)+0.58\,(0,1)+0.28\,(0.5,0.5)=(0.28,\,0.72), mostly cat's value

Return to contents

Slide 32

Try it: attention, step by step

Return to contents

Slide 33

Try it: move the query

Return to contents

Slide 34

One sentence, one matrix of weights

A six-by-six grid of attention weights for the sentence the cat sat on the mat, rows the word being updated, columns the word being read, where every word can read every word: sat reads cat most, on reads sat and mat. The space on the right, where the masked version appears on the next slide, is empty

A six-by-six grid of attention weights for the sentence the cat sat on the mat, rows the word being updated, columns the word being read, where every word can read every word: sat reads cat most, on reads sat and mat. The space on the right, where the masked version appears on the next slide, is empty

Open full-size figure

Schematic: hand-set scores, softmax computed · every row sums to 1

Return to contents

Slide 35

The causal mask: no peeking ahead

Two six-by-six grids of attention weights for the sentence the cat sat on the mat, rows the word being updated, columns the word being read. Left, every word can read every word. Right, the causal mask: every cell above the diagonal is hatched and holds zero, so each word reads only itself and the words before it

Two six-by-six grids of attention weights for the sentence the cat sat on the mat, rows the word being updated, columns the word being read. Left, every word can read every word. Right, the causal mask: every cell above the diagonal is hatched and holds zero, so each word reads only itself and the words before it

Open full-size figure

Schematic: hand-set scores, softmax computed · every row sums to 1

Return to contents

Slide 36

Back to the bigram table

Back to the bigram table — table
Model Context it uses Held-out loss (nats) Effective choices, elosse^{\text{loss}}
Uniform guess none 4.57 97
Bigram table the current character 2.28 9.8
Average every earlier character, equal weights 2.03 7.6
One attention head every earlier character, chosen weights 1.23 3.4

Return to contents

Slide 37

Recap

Return to contents

Slide 38

Further reading for the research trail

Optional: a few research-trail entries over the semester. Any paper mentioned in any lecture qualifies, but papers from these lectures are preferred.

Return to contents

Slide 39

Minute paper

Submit on Canvas → Minute papers → Minute Paper 16 (access code read out in class).

Write three brief points in your own words:

  1. Averaging the past weighs every earlier token the same. In your own words, how does attention let a token choose which earlier tokens to read?
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents