Large language models and the brain

Reader-friendly version of the 55 lecture slides: the lecture content in normal flow, with figure descriptions and data tables where the figure carries data. Open the presentation.

Slide 1

Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 17 · Theme 3: Dynamics, sequences & world models

Large language models and the brain

Tuesday, November 17 · Fall 2026

Return to contents

Slide 3

Finish the sentence

He spread the warm bread with ___
  • Most of you thought butter: you predicted the next word, as our language models do
  • socks (from the attention lecture) → larger EEG response ~400 ms after the word: the N400
  • A language model assigns every word a surprisal (how unexpected it is). Does surprisal predict reading times and N400 size?
Kutas and Hillyard 1980, Fig. 1A: the average EEG at the back of the scalp while people read three seven-word sentences one word per second, plotted with negative up. The traces are similar until the last word. It was his first day at work, solid, ends flat; He spread the warm bread with socks, dashed, gives a large negative peak about 400 ms after socks, labeled N400; She put on her high heeled SHOES, with the last word in large type, dotted, gives a positive peak labeled P560

Published figure: Figure 1A of Kutas and Hillyard (1980), Science. Average EEG recorded at the back of the scalp while people read three seven-word sentences, one word per second. The horizontal axis is time, with the onset of each word marked by an arrow and the words printed below; a bracket marks 1.0 second. The vertical axis is voltage, with a 10 microvolt scale bar and negative plotted upward. Three traces are drawn, one per sentence: a solid line for It was his first day at work, a dashed line for He spread the warm bread with socks, and a dotted line for She put on her high heeled SHOES, whose last word is printed in large type. The three traces are similar through the first six words. After the last word, the solid trace stays near baseline. The dashed trace swings to a large negative peak about 400 ms after socks, labeled N400. The dotted trace swings the other way, to a positive peak labeled P560. The slide uses this to show that a word that does not fit its sentence produces a larger N400, the response that surprisal is later compared with.

Open full-size figure

Return to contents

Slide 4

Scaling up the language model

The same transformer, trained on more text than anyone could read

Return to contents

Slide 5

Where we left off

Return to contents

Slide 6

Scaling laws: the loss falls smoothly with size

Figure 1 of Kaplan et al.: three log-log panels of test loss against compute, dataset size and parameters, each a straight line, a power law, over several orders of magnitude; the fitted exponents are minus 0.050, minus 0.095 and minus 0.076

Published figure: Figure 1 of Kaplan et al. (2020), Scaling laws for neural language models. Three side-by-side line plots, each with both axes on a logarithmic scale and the language model's test loss (cross-entropy, from about 2 to 7) on the vertical axis. Left panel: loss against compute in petaflop-days, from 10 to the minus 9 up to 10, drawn as many pale blue training curves with a black lower envelope and an orange dashed power-law fit, loss = (compute / 2.3 times 10 to the 8) to the power minus 0.050. Middle panel: loss against dataset size in tokens, from about 2 times 10 to the 7 to 2 times 10 to the 9, with six blue points falling from about 4.1 to 2.7 along a grey line with exponent minus 0.095. Right panel: loss against non-embedding parameters, from about 10 to the 3 to 10 to the 9, with many points falling from about 5.6 to 2.3 along a grey line with exponent minus 0.076. On a log-log plot a power law is a straight line, and all three are straight over several orders of magnitude. The slide uses this to show that loss falls smoothly and predictably as compute, data or parameters grow.

Open full-size figure

Return to contents

Slide 7

From characters to word pieces

Return to contents

Slide 8

Word pieces in GPT-2

Return to contents

Slide 9

How much text do modern LLMs use?

Data, the first ingredient of scale, counted in tokens: how much do models read?

T = trillion · B = billion · M = million · web figures are estimates

Return to contents

Slide 10

Model size: the largest are undisclosed

Model size: the largest are undisclosed — table
Model Year Built by Training tokens Parameters
GPT-2 2019 OpenAI ~0.01 T 0.1 – 1.5 B
GPT-3 2020 OpenAI 0.3 T 175 B
Chinchilla 2022 DeepMind (Google) 1.4 T 70 B
Llama 3 2024 Meta 15 T 8 – 405 B
Frontier models 2023 on OpenAI, Google, … ? ?

Return to contents

Slide 11

Compute: the third ingredient

Scatter plot from Epoch AI's dataset of notable AI models: training compute in floating-point operations, log scale from 10 to the 16 to 10 to the 28, against publication year, 2017 to 2026, for 286 language models. A fitted line rises about 6.2 times per year. Labeled: GPT-2 in 2019 at about 2 times 10 to the 21; GPT-3 in 2020 at about 3 times 10 to the 23; Chinchilla in 2022 at about 6 times 10 to the 23; GPT-4 in 2023 at about 2 times 10 to the 25; Gemini 1.0 Ultra in late 2023 at about 5 times 10 to the 25; Llama 3.1-405B in 2024 at about 4 times 10 to the 25; and GPT-6 Astra in 2026, the largest, at about 9 times 10 to the 26

Figure computed for this lecture from Epoch AI's Notable AI Models dataset (CC-BY 4.0, downloaded 2026-11-17), language models only. A scatter plot. The horizontal axis is publication year, 2017 to 2026. The vertical axis is training compute in floating-point operations (FLOP), on a logarithmic scale with gridlines at 10 to the 16, 19, 22, 25 and 28. Each small blue dot is one model, 286 in all. A magenta line is the least-squares fit of log compute against date for the 276 models from 2018 on; it rises about 6.2 times per year, a doubling every 4.6 months. Seven models are labeled with larger dots: GPT-2 (largest) in 2019 at about 2 times 10 to the 21, GPT-3 175B in 2020 at about 3 times 10 to the 23, Chinchilla in 2022 at about 6 times 10 to the 23, GPT-4 in 2023 at about 2 times 10 to the 25, Gemini 1.0 Ultra in late 2023 at about 5 times 10 to the 25, Llama 3.1-405B in 2024 at about 4 times 10 to the 25, and GPT-6 Astra in 2026 at about 9 times 10 to the 26, the highest point. The compute of the frontier models is Epoch's estimate, since the labs do not publish it. From GPT-2 to GPT-6 Astra, training compute grows about 500,000 times. The slide uses this to show compute as the third ingredient, after data and parameters, in the growth of language models.

Open full-size figure

Return to contents

Slide 12

Scaling laws in practice

Data, parameters, compute: how should a fixed compute budget be split between model size and data?

Return to contents

Slide 13

Do large models converge?

As models scale, do they also become alike?

Return to contents

Slide 14

In-context learning: what a prompt looks like

Return to contents

Slide 15

In-context learning

Accuracy against the number of examples in the prompt, from zero-shot through one-shot to few-shot, for three GPT-3 sizes. The 175-billion-parameter model climbs from under to about 67%; the 13-billion model reaches about and the 1.3-billion model stays below 5%. Solid lines add a natural-language description of the task, dashed lines do not

Published figure from Brown et al. (2020), the GPT-3 paper, showing accuracy on a word-unscrambling task against the number of examples placed in the prompt. The horizontal axis is the number of examples in context, 0 at the left, then 1 and 10 on a logarithmic scale; labels above mark zero-shot (no examples), one-shot (one example) and few-shot (many). The vertical axis is accuracy in percent, 0 to about 67. Three colored groups of lines are labeled 175B, 13B and 1.3B parameters; for each size a solid line is a prompt that also describes the task in natural language and a dashed line has no such description. The 175-billion-parameter model starts at about 8 percent with no examples, reaches about 46 percent with one example when the task is described (about 26 without), and climbs to about 67 percent. The 13-billion model rises from about 1 to about 25 percent, and the 1.3-billion model stays below 5 percent. The task description helps mostly when examples are few. The slide uses the figure to show that in-context learning improves with model size and with more examples.

Open full-size figure

In the figure: solid = prompt opens with a one-sentence instruction ("Please unscramble the letters…"); dashed "No Prompt" = examples only, no instruction

Return to contents

Slide 16

Our replication: in-context learning in GPT-2

Antonyms correct in percent against the number of solved examples in the prompt, 0, 1, 2, 4 and 8, for GPT-2 small, medium and large. Small stays below 10 percent at every count. Medium rises from 0 to 22, 64, 88 and 94 percent; large from 4 to 25, 81, 92 and 96 percent

Real computation on three small public models, GPT-2 small (124 million parameters), medium (355 million) and large (774 million). A line plot with solid points joined by lines. The horizontal axis is the number of solved examples in the prompt, at 0, 1, 2, 4 and 8. The vertical axis is the percentage of antonym questions answered correctly, 0 to 100. The prompt is a list of word pairs such as hot, cold, followed by a new word whose opposite the model must produce. GPT-2 small stays low at every count, peaking at about 10 percent with one example and sliding to about 4 percent at eight. GPT-2 medium rises from 0 to about 22, 64, 88 and 94 percent. GPT-2 large rises from about 4 to 25, 81, 92 and 96 percent. The two larger models nearly coincide by eight examples. No weights change in any of these runs; only the prompt grows. The slide uses this to show that in-context learning appears with scale and improves with examples, using models students can run themselves.

Open full-size figure

Computed for this course: GPT-2 small, medium and large, weights as trained · antonym task · 32 pairs, 24 test words × 3 random orders per point

Return to contents

Slide 17

In-context learning: surprising results

Beyond accuracy curves: what does the model take from the examples?

Return to contents

Slide 18

Where in-context learning comes from

Nobody trained for this ability: where does it come from?

Figure 1.1 of Brown et al.: an outer loop, learning by stochastic gradient descent during pretraining, runs across many text sequences; within each sequence an inner loop of in-context learning runs down a list of examples of one task: additions such as 5 + 8 = 13, unscrambled words such as gaot => goat, and English to French translations such as thanks => merci

Published diagram: Figure 1.1 of Brown et al. (2020), the GPT-3 paper, a schematic of language-model meta-learning with two nested loops. The outer loop is learning by stochastic gradient descent during pretraining, which runs over many text sequences. Inside each sequence is an inner loop of in-context learning, a list of examples of one task followed by a new case the model must complete. Three example sequences are shown: addition, such as 5 + 8 = 13 and 7 + 2 = 9, then a new sum; unscrambling words, such as gaot => goat, then a new scrambled word; and translating English to French, such as thanks => merci, then a new word. The point is that the model learns, across training, to pick up a task from the examples in its own context. The slide uses this to name in-context learning as a skill acquired by pretraining on text, with no weight update at test time.

Open full-size figure

Return to contents

Slide 19

Two loops during pretraining

Ordinary next-token training, described as two loops:

Return to contents

Slide 20

Mechanism 1 (last lecture): induction heads copy

Two bands of word boxes for The black cat sat on the mat. The black, the last black marked as the current position. Top band, block 1, a previous-token head: an arrow from each word to the next, and the first cat receives a tag reading previous: black. Bottom band, block 2, an induction head: the query of the last black, find previous = black, matches the key of the first cat, previous: black, along a curved gold arrow; the value of cat, cat, flows to a box reading predict: cat. A faint dashed arrow from the last black back to the first black is labelled one head alone: finds black, reads back black

Diagram drawn for this lecture: how two attention heads in two successive blocks of a transformer predict the next word of The black cat sat on the mat. The black. Two horizontal bands each show the same ten tokens as boxes, with the last token, black, marked as the current position. The top band, block 1, is a previous-token head: each position reads the token before it, and the first cat receives a tag reading previous: black. The bottom band, block 2, is an induction head. The last black issues a query, find: previous = black. This query matches the key on the first cat, previous: black, along a curved gold arrow. The value on that cat, cat, is read and goes to a box reading predict: cat. A faint dashed arrow from the last black back to the first black is labeled one head alone: finds black, reads back black. It shows what one head alone would do: it finds the earlier black and returns black. The slide uses this to show that an induction head copies the word that came after an earlier occurrence of the current word, and that it needs the tag written by the previous-token head in the block before.

Open full-size figure

Return to contents

Slide 21

Mechanism 2: heads that encode the task

Figure 1a of Yin and Steinhardt: few-shot in-context-learning accuracy of Pythia 6.9B as a growing percentage of its attention heads is ablated. Clean and random-ablation lines stay flat near 0.56; ablating induction heads lowers accuracy slowly to 0.44 at 20 percent; ablating function-vector heads drops it to about 0.32 at 3 percent, 0.20 at 9 percent and 0.18 at 15 percent

Published figure: Figure 1a of Yin and Steinhardt (2025), Which attention heads matter for in-context learning?, ICML 2025. A line plot titled Pythia 6.9B. The horizontal axis is ablation, the percentage of the model's attention heads that are knocked out, from 1 to 20 percent. The vertical axis is in-context learning accuracy on few-shot tasks, from about 0.15 to 0.6. Four lines are drawn. Clean, in yellow, is the intact model and stays flat at about 0.56. Random, in green, knocks out randomly chosen heads and stays near 0.55, ending at about 0.54 at 20 percent. Induction, in blue with round markers, knocks out induction heads. Accuracy falls slowly, from about 0.55 at 1 percent to about 0.44 at 20 percent. FV, in pink with crosses, knocks out function-vector heads. Accuracy falls quickly, from about 0.46 at 1 percent to about 0.32 at 3 percent, about 0.20 at 9 percent and about 0.18 at 15 and 20 percent. The slide uses this to show that function-vector heads matter much more for learning a task from examples than induction heads do.

Open full-size figure

In the figure: ICL = in-context learning · FV = function-vector heads · ablation = knockout

Return to contents

Slide 22

Mechanism 3: learning without backprop

Third account: can a forward pass itself carry out a learning algorithm?

Return to contents

Slide 23

The same recipe in real systems

What a 2026 model is, and what every prompt does

Return to contents

Slide 24

From GPT-2 to today's assistants

Same recipe, GPT-2 to today: what changed?

Return to contents

Slide 25

From autocomplete to assistant

Figure 2 of Ouyang et al.: three steps. Step 1, a labeler writes a good answer to a prompt such as Explain the moon landing to a 6 year old, and GPT-3 is fine-tuned on these demonstrations. Step 2, the model writes several answers to a prompt, a labeler ranks them from best to worst, and the rankings train a reward model. Step 3, for a new prompt such as Write a story about frogs, the model writes an answer, the reward model scores it, and the score updates the model by reinforcement learning

Published diagram: Figure 2 of Ouyang et al. (2022), the InstructGPT paper, showing three training steps as three panels from left to right. Step 1, supervised fine-tuning: a prompt such as Explain the moon landing to a 6 year old is given to a human labeler, who writes a good answer; GPT-3 is fine-tuned on these demonstrations. Step 2, reward model: the model writes several answers to a prompt, a labeler ranks them from best to worst, and the rankings train a reward model that scores answers. Step 3, reinforcement learning: for a new prompt such as Write a story about frogs, the model writes an answer, the reward model scores it, and that score updates the model with a policy-gradient method (PPO). Arrows connect prompts, outputs, labelers and models. The slide uses it to show how a pretrained next-token predictor becomes an assistant that follows instructions.

Open full-size figure

Return to contents

Slide 26

After InstructGPT: simpler and checkable rewards

RLHF is costly: later post-training drops the reward model, then the human labels.

Return to contents

Slide 27

Writing an answer is choosing a path

Why would writing out steps help a model answer?

Illustrative tree of token choices for the prompt 17 times 24 equals. The top path takes the most likely next token at every step, 3, 9, 8, and ends in 398, marked wrong. Three other paths write intermediate steps: 17 times 20 is 340, 17 times 4 is 68, 340 plus 68 is 408; 24 times 17, 24 times 10 is 240, 24 times 7 is 168, 240 plus 168 is 408; and 17 times 25 is 425, 425 minus 17 is 408. All three end in 408, marked correct. Faint grey stubs branch off the nodes, showing other possible continuations

Diagram drawn for this lecture, labeled Illustrative: a tree of token choices for the prompt 17 times 24 equals. The prompt is a box at the left. Four paths leave it, each a row of boxes joined by arrows and ending in a box at the right. The top path, labeled most likely next token, every step, takes the single most likely token each time: 3, then 9, then 8. It ends in a magenta box, 398, marked wrong. The three paths below, labeled many different paths reach the goal, write intermediate steps in green boxes. The first reads 17 times 20 = 340, then 17 times 4 = 68, then 340 + 68 = 408. The second reads 24 times 17, then 24 times 10 = 240, then 24 times 7 = 168, then 240 + 168 = 408. The third reads 17 times 25 = 425, then 425 - 17 = 408. All three end in a green box, 408, marked correct. Faint grey stubs branch off several boxes and stand for other continuations the model could have chosen. The probabilities of the choices are not from a real model. The slide uses this to show that an answer is a path through a large tree of continuations, and that choosing the most likely token at each step can end in a wrong answer.

Open full-size figure

Return to contents

Slide 28

Reasoning: thinking in tokens

Two ways to get intermediate steps: prompt for them, or train for them.

Return to contents

Slide 29

Inside a 2026 model

Same pretrained core, seen from inside: how do models grow without cost growing with them?

Return to contents

Slide 30

Every prompt is in-context learning

Back to the prompt: what does an assistant read before it answers?

Return to contents

Slide 31

Tools and agents

The context can also hold tool results: the model's text triggers actions.

Return to contents

Slide 32

A model of the reader, and of the brain?

Two tests of the same model: word probabilities → reading times and N400; internal activations → brain responses

Return to contents

Slide 33

Surprisal: how unexpected a word is

First test: do a model's word probabilities predict readers and the N400?

surprisal(wi)=−log⁡2p(wi∣w1,…,wi−1) bits\mathrm{surprisal}(w_i)=-\log_2 p\big(w_i\mid w_1,\dots,w_{i-1}\big)\ \text{bits}

Return to contents

Slide 34

Surprisal from a real model, token by token

What does surprisal look like from a real model?

Bar chart of surprisal in bits for each token of the sentence My grandmother's recipe calls for two cups of flour and a pinch of salt, three bars per token for GPT-2 small, medium and large. Grandmother is the most surprising token, 14.5 bits for the small model and about 8.5 for the larger two; recipe about 8 to 10 bits; for and the second of are near zero for every model; salt is near zero. Means 4.0, 3.5 and 3.3 bits

Real computation on three public models, GPT-2 small (124 million parameters), medium (355 million) and large (774 million). A grouped bar chart. The horizontal axis lists the tokens of the sentence My grandmother's recipe calls for two cups of flour and a pinch of salt, with a leading space marked by a dot; the first token has no bar because nothing precedes it. Each token has three bars, one per model, giving its surprisal in bits, the negative base-2 log probability the model gave it given the earlier words, on a vertical axis running 0 to about 15. Grandmother is the tallest, 14.5 bits for the small model (a callout says about 1 chance in 22,657) and about 8.5 for the medium and large ones. Recipe is about 8 to 10 bits. The token for and the second of are near zero, and a callout notes that 0 bits means the model was sure. Salt is also near zero. Mean surprisal over the sentence is 4.0, 3.5 and 3.3 bits for small, medium and large. The slide uses this to show what surprisal is and that bigger models are on average less surprised.

Open full-size figure

Real computation: GPT-2 small, medium and large, weights as trained · a dot marks a token that begins with a space

Return to contents

Slide 35

What it means to predict reading times

From per-word surprisal to a test against readers' behavior.

Two aligned bar charts for the sentence Seeds from a tulip will form a flowering bulb after seven-twelve years: mean reading time per word on top, from 314 to 873 ms, and GPT-2 small surprisal per word below. The word seven-twelve has the largest surprisal, 24.8 bits, and a slow reading time of 585 ms; the next word, years, is read slowest, 873 ms, at only 8.4 bits

Real data and computation on one sentence from the Natural Stories self-paced reading corpus, Seeds from a tulip will form a flowering bulb after seven-twelve years. Two stacked bar charts share the same twelve words along the bottom. The top chart gives mean reading time in milliseconds, 0 to about 900, averaged over many readers. Most words sit between about 310 and 500 ms (Seeds about 490, a about 310). Seven-twelve, drawn in red, is about 580 ms, and the final word years is the slowest at about 870 ms. The bottom chart gives each word's surprisal in bits under GPT-2 small, 0 to about 25. Most words are 2 to 10 bits (flowering about 10, the second a about 2), while seven-twelve, also in red, is the clear peak at about 25 bits. The unexpected word draws the highest surprisal and a longer reading time, and the slowdown carries into the next word. The slide uses this to link model surprisal to human reading times.

Open full-size figure

Real computation: Natural Stories self-paced reading times · GPT-2 small surprisal, whole story as context

Return to contents

Slide 36

Reading time against surprisal, the whole story

Scatter plot of mean reading time in milliseconds against GPT-2 small surprisal in bits for the 1,038 words of one Natural Stories story. Most words have surprisal under 15 bits and reading times between 300 and 450 ms. A line of binned means rises slowly from about 340 ms to about 370 ms up to 17 bits, then to about 570 ms near 28 bits. Two words are highlighted: seven-twelve at 24.8 bits and 585 ms, and years at 8.4 bits and 873 ms. Text in the corner: r = 0.30, n = 1038 words

Real data and computation on one story of the Natural Stories self-paced reading corpus. A scatter plot with one dot for each of the 1,038 words of the story. The horizontal axis is the word's surprisal in bits under GPT-2 small, with the whole story as context, from 0 to about 35. The vertical axis is the mean reading time of the word in milliseconds, averaged over readers, from 200 to about 1,000. Most dots are blue and pale, and they pile up at surprisal below 15 bits and reading times between about 300 and 450 ms. A dark line joins the means of the dots in successive surprisal bins. It rises slowly from about 340 ms near 2 bits to about 370 ms near 17 bits, then climbs to about 570 ms near 28 bits, and ends lower, at about 480 ms near 32 bits, where few words fall. Two words are drawn in red. Seven-twelve is at 24.8 bits and about 585 ms. Years is at 8.4 bits and about 873 ms, far above the line for its surprisal. Text in the corner reads r = 0.30, n = 1038 words. The slide uses this to show that more surprising words are read more slowly, with a lot of scatter, and that surprisal explains about 9 percent of the variance in reading time.

Open full-size figure

Real computation: Natural Stories, one story · GPT-2 small, whole story as context · dot = one word · line = mean per bin

Return to contents

Slide 37

Against human readers: bigger is not better

Scatter plot of regression log-likelihood gain against perplexity on Natural Stories self-paced reading, for GPT-2 small to XL, five GPT-Neo and eight OPT models with dashed least-squares lines per family. Every family slopes the same way: the smallest models, with the highest perplexity, give the largest gain near 600 to 640, and the largest models, 20 to 66 billion parameters, the smallest, near 380 to 400

Published figure: Figure 1 of Oh and Schuler (2023), panel for the Natural Stories self-paced reading corpus. A scatter plot. The horizontal axis is language-model perplexity on a log scale, from about 2 to the 3.2 to 2 to the 4.8; the vertical axis is delta log-likelihood, the gain in how well a regression predicts human reading times when the model's surprisal is added, from about 375 to 640. Three model families are plotted with dashed least-squares lines: GPT-2 (blue circles, small to XL), GPT-Neo (orange squares, 125 million to 20 billion parameters) and OPT (green hexagons, 125 million to 66 billion). Each point is labeled with its size. In every family the smallest models, with the highest perplexity, give the largest gain, near 600 to 640, and the largest models give the smallest, near 380 to 400. Better next-word predictors therefore explain reading times worse. The slide uses this to show that the best language model is not the best model of human reading.

Open full-size figure

Each dot = one model (label = size) · left = lower perplexity, better text prediction · up = better fit to reading times · in every family, bigger models sit lower

Return to contents

Slide 38

Try it: what did the model expect?

Return to contents

Slide 39

Better language model, better brain model?

Reading times disagree with scale: what about brain responses?

Return to contents

Slide 40

Using the model to control the brain

Figure 2 of Tuckute et al.: A, the left-hemisphere language network on an inflated brain; B and C, bar charts of the language network's fMRI response to drive, suppress and baseline sentences, in an event-related design with three participants and a blocked design with four. Drive sentences give the highest response and suppress sentences the lowest in both; D, example sentences: drive sentences are odd or ungrammatical, such as Jiffy Lube of, of therapies, yes; suppress sentences are plain, such as We were sitting on the couch

Published figure: Figure 2 of Tuckute et al. (2024), driving and suppressing the human language network with sentences chosen by a language model. Panel A shows the left hemisphere on an inflated brain, with the language network in red along the temporal lobe and in frontal cortex, defined separately for each participant. Panel B is a bar chart for an event-related design with three participants. Vertical axis: z-scored fMRI (BOLD) response of the language network; bars for the drive, suppress and baseline conditions. Drive sentences are highest at about 0.28, suppress sentences are lowest at about minus 0.29, and baseline is near 0; dots show individuals. Panel C is the same for a blocked design with four participants: drive about 0.7, suppress about minus 1.2, baseline about 0.5. Panel D lists example sentences. Drive sentences are odd or fragmentary, such as Jiffy Lube of -- of therapies, yes. Suppress sentences are plain, such as We were sitting on the couch. Baseline sentences are ordinary corpus text. The slide uses this to show that a model of language can predict, and control, activity in the brain's language regions.

Open full-size figure

Return to contents

Slide 41

The brain as a prediction machine

Return to contents

Slide 42

Open questions

Return to contents

Slide 43

Recap

Return to contents

Slide 44

The assistant you may use on the project

Return to contents

Slide 45

For your final project

Return to contents

Slide 46

Further reading

Papers from today worth reading, for your project or out of interest.

Return to contents

Slide 47

Minute paper

Submit on Canvas → Minute papers → Minute Paper 18 (access code read out in class).

Write three brief points in your own words:

  1. A model answers a new task correctly after reading three solved examples in its prompt. What changed inside the model, and what did not? How is this different from fine-tuning?
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents

Slide 48

Going further

Predictive coding in detail

Return to contents

Slide 49

Language models and the brain: current readings

More encoding-model results, from conversation to single neurons, with a caution.

Return to contents

Slide 50

Testing it with a language model

Figure 2 of Caucheteux et al.: a, the brain score of GPT-2 activations on fMRI from 304 listeners, highest along the superior temporal sulcus; b and d, adding the model's representation of words d positions ahead improves the brain score, with a peak at about 8 words ahead; c, the regions where this forecast gain is significant, temporal, parietal and frontal; e to g, the best forecast distance grows from temporal regions (about 8 words) to frontal and parietal regions (about 9 to 10 words)

Published figure: Figure 2 of Caucheteux, Gramfort and King (2023), evidence for predictive coding in the brain during language comprehension. Seven panels labeled a to g, from fMRI of 304 people listening to stories. Panel a, brain maps: the brain score of GPT-2 activations, how well they predict each region's activity, from 0.005 to 0.230, is highest along the superior temporal sulcus. Panels b and d, line plots: adding the model's representation of the word d positions ahead (horizontal axis, distance to the current word, minus 10 to 30) raises the brain score, peaking at about 8 words ahead. Panel c, brain map: the regions where this forecast gain is significant, across temporal, parietal and frontal cortex. Panel e, brain map: the best forecast distance in each region, from 7.5 to 9.5 words. Panel f, line plots comparing two regions: forecast score peaks earlier in anterior superior temporal sulcus (about 7 words) than in inferior frontal gyrus (about 9). Panel g, dots by region, show best forecast distance growing from about 8 in the mid superior temporal sulcus to about 9.2 in frontal regions. The slide uses this to show that the brain appears to predict upcoming words, further ahead in higher regions.

Open full-size figure

Return to contents

Slide 51

Predictive coding and autoencoders

If the brain minimizes prediction error, how does that compare with an autoencoder?

Return to contents

Slide 52

Rao & Ballard (1999)

Return to contents

Slide 53

The predictive-coding equations

At one level: prediction of the level below 𝐱^=𝐖𝐫\hat{\mathbf{x}} = \textcolor{#B5396B}{\mathbf{W}}\mathbf{r}, error 𝐞=𝐱−𝐱^\mathbf{e} = \mathbf{x} - \hat{\mathbf{x}}.

Inference (fast, within one stimulus: settle 𝐫\mathbf{r} to reduce the error):

𝐫←𝐫+ηr 𝐖⊤𝐞\mathbf{r} \leftarrow \mathbf{r} + \eta_r\,\textcolor{#B5396B}{\mathbf{W}}^\top\mathbf{e}

Learning (slow, across stimuli: a local, Hebbian-like rule):

𝐖←𝐖+ηW 𝐞 𝐫⊤\textcolor{#B5396B}{\mathbf{W}} \leftarrow \textcolor{#B5396B}{\mathbf{W}} + \eta_W\,\mathbf{e}\,\mathbf{r}^\top

Return to contents

Slide 54

Predictive coding as a deep net: PredNet

  • Lotter, Kreiman & Cox stacked Rao–Ballard-style modules: each layer predicts the layer below and passes the prediction error up, trained on video to predict the next frame
  • The trained network reproduces perceptual effects nobody built in, such as the flash-lag illusion
  • From brain to AI: a 1999 theory becomes a trainable architecture
The PredNet module: representation units emit a prediction of the layer below, the mismatch becomes an error signal, and errors propagate up the hierarchy

Published diagram: Figure 1 of Lotter, Kreiman and Cox (2016), the PredNet architecture. Left side: two stacked layers, l and l plus 1. Each layer has four kinds of square module. Green representation units R, with a recurrent loop arrow, send a prediction down to a blue module (A-hat), the predicted input. This is compared, through a circled minus sign, with a blue target module A, the actual input of that layer. The difference feeds a red error module E. Error signals go up the hierarchy to the next layer's target and to the representation units, and the representation of the higher layer passes downward to the one below. Right side: the same unit drawn with its operations. Representation is a convolutional LSTM; the prediction is a convolution; the target is a convolution then pooling of the input; and the error subtracts the two and applies a rectifier to the positive and negative parts. The slide uses this to show a network built around predicting its input and passing on only the error.

Open full-size figure

Return to contents

Slide 55

Is backprop biologically plausible?

Return to contents