
The two centers in each column are physically identical
Optional: for going deeper, or as a starting point for the final project.
Submit on Canvas → Minute papers → Minute Paper 19 (access code read out in class).
Write three brief points in your own words:
Credit for a thoughtful attempt, not for being correct.
- The last three lectures built a transformer and scaled it into a large language model: at every step, attention looks back at the stored keys and values of every earlier token - Today: the other way to use the past. A recurrent network keeps one running summary, its state, and updates it at every step - Five parts: what a state is; how to build a recurrent unit and why it is hard to train; what recurrence buys in visual cortex; why recurrence is back inside 2025-26 language models; loops over depth
- A feedforward network sees only its current input. Next slide: two ways a network can use the past - Whatever is kept, the past has to enter the computation somehow - Figure, three tasks where the current item does not fix the answer. Language: "bank" means a river edge in "the river bank" and a financial institution in "the money in the bank". Vision: a ball moves behind a screen; in the middle frame nothing is visible, and the ball reappears on the right. Arithmetic: 478 + 256 = 734, where the carries come from the columns to the right - The vision networks of earlier lectures had no memory: each image was processed alone - A transformer takes the first route: its input is the whole context window and it computes the answer in one feedforward pass. This lecture takes the second route: a small input and a state carried from step to step
- Both panels do the same job: read "the cat sat on the" and predict the next word, "mat". First the transformer's way - Every earlier word keeps a key and a value. The current word sends a query that reads all of them, then the prediction is made. When the next word arrives, the whole sequence is read again - The KV cache stores the keys and values so they are not recomputed, but each new word still reads every one of them: memory and work grow with the length of the text
- Now the recurrent way, on the same words - Right, recurrence: one input at a time. Each word updates the state; the last state alone makes the prediction. Earlier words are never read again
- The superscript ⟨t⟩ is the time step: h⟨t⟩ is the state after the input at step t. It is not a power and not a layer index (layers use [l]) - Two operations define any system with a state: the update combines the old state with the new input; the readout computes the output from the state alone - The past acts on the future only through the state: given h⟨t−1⟩ and x⟨t⟩, earlier inputs add nothing - The first three rows are the tasks of the opening slide: the state is exactly what the current item does not show - Last two rows: biology vs. artificial networks. In the brain, the state is the activity of the neurons plus slow variables (membrane voltage, the state of ion channels and synapses) that let past input persist, so a new input changes them instead of replacing them. In an artificial network, the state is just the vector of unit activities
- In a network, what to keep is learned: it is set by the weights that mix the old state with the new input (the unit we build later today) - A state of fixed size cannot hold a past of any length, so something is lost - Sachs (1967): after hearing a story, people judged whether a test sentence was identical to one in the story. The test came right away, or after about 80 or 160 more syllables. Changed versions either kept the meaning (active to passive, a small change of form) or changed it (who did what to whom) - Right away, both kinds of change are detected. After about 80 syllables, changes of wording go unnoticed and changes of meaning are still detected - Whatever a state drops is gone, with no way back to re-read it. Attention keeps every token and can look back - Sachs, J. S. (1967). Recognition memory for syntactic and semantic aspects of connected discourse. Perception & Psychophysics
- Here is a state measured in a brain, in a task that forces the animal to hold one fact over a delay - Figure, panel A: timeline of the task. Intertrial interval, fixation (F), cue (C), delay (D) with only the fixation point on the screen, response (R), then reward. Panel B: one trial. The horizontal and vertical gaze traces stay flat from fixation to the end of the delay and jump at the response; below, the spikes of one prefrontal neuron are dense through the whole delay - The gaze stays on the fixation point until the response, so the monkey cannot solve the task by looking at the cued place during the delay - The spikes in the intertrial interval are the neuron's background firing; the paper measures delay activity against this ITI rate. A single trial cannot show selectivity: that needs all eight cue locations (next slide) - Funahashi et al. recorded 288 neurons around the principal sulcus; 87 changed their firing during the delay - Funahashi, S., Bruce, C. J., & Goldman-Rakic, P. S. (1989). Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. Journal of Neurophysiology
- This is the same neuron population, now across the eight cue locations - Figure: the cue locations (0, 45, 90, 135, 180, 225, 270, 315 degrees) sit around a central diagram. Each panel shows spike rasters of single trials with their average below; vertical lines mark cue (C), delay (D) and response (R). Scale bars: 50 spikes per second and 1 s - Only the panel for the cue at 270 degrees, at the bottom, shows dense firing through the whole 3 s delay. For cues in the upper field (45, 90, 135 degrees) delay firing is significantly below the ITI rate: the neuron is not firing to everything, it is selective in both directions - The firing did not depend on the length of the delay (1 to 6 s). Of the 87 neurons with delay activity, 79% were directional like this one - The location is held in activity, with no stored copy of the image - Zipser (1991) trained a small recurrent network to hold a value over a delay, as in this task; its hidden units developed delay activity like that recorded in monkey cortex. Many later models (e.g., Yang et al. 2019, Nature Neuroscience) train recurrent networks on this and related tasks and compare their units with prefrontal neurons - Zipser, D. (1991). Recurrent network model of the neural mechanism of short-term active memory. Neural Computation - same paper as the previous slide (Funahashi, Bruce, & Goldman-Rakic, 1989, Journal of Neurophysiology)
- The model neuron so far: a weighted sum of its current inputs, then an activation function. Its output depends on the current input only - Four steps follow: an integrator, a leak, a network of such units with weights, and training through time
- One number - Figure (computed for this slide), time steps 1 to 40. Top: the input x, with pulses of +1 at steps 5, 6, 7 and 15 and of -1 at steps 25 and 26, zero elsewhere. Bottom: the running total h, a step curve that climbs to 3 after the first three pulses, to 4 after step 15, drops to 2 after the two negative pulses and stays at 2 until step 40 - This is the simplest recurrent unit: one number, updated at every step from its old value and the new input - It remembers perfectly, which is also its problem: an old input counts as much as a new one, and the state can grow without bound
- The leak fixes the two problems, no forgetting and unbounded growth - Figure: the same input and running total, plus the green leaky state with lambda = 0.8. It rises to 2.44 at step 7, decays toward zero between pulses (0.46 at step 20), dips below zero after the negative pulses (-0.69 at step 30) and is back near zero (-0.07) at step 40, while the running total still holds 2 - With no input, the state shrinks by the factor lambda each step: after k steps, lambda^k of it is left - Link to the time constant: with steps of Δt, λ ≈ 1 − Δt/τ (the fraction left after one step); e.g., 20 ms steps and τ = 100 ms give λ = 0.8, the value in the figure - The time constant tau is the time in which a neuron's activity loses about two-thirds of its distance to rest; it ties the network's steps to milliseconds, so it can be compared with a brain or a reaction time - lambda is a parameter of the unit, like a weight, so it is drawn in the weight color
- Now one number becomes a layer of units, built in four steps (this slide, then the state, the recurrent connections and the readout) - Figure: the input vector x at step t, drawn as four bars, sends an arrow labeled W_hx up into a box of four units - Subscripts on a weight matrix read destination from source: W_hx goes to h from x - The input is the same kind of vector as in every network so far; the bars stand for its entries
- The units add up the weighted input and apply a nonlinearity - Figure: the same diagram; the units now carry the label h at step t (their activities), and a small g marks the nonlinearity - g is usually tanh, the S-shaped activation running from -1 to 1; b_h is a bias - With only this much, the layer is the model neuron of the earlier lectures many times over: a weighted sum and a nonlinearity
- Figure: green arrows are recurrent connections, magenta arrows feedforward ones. Every unit connects to itself; the arrows between units are drawn for h1 only, to keep the figure readable, but every unit receives them - One added connection turns it into a network with a state - Figure: the same diagram with recurrent connections. An arrow leaves the units on the right, labeled h at step t-1, and comes back in on the left through the weights W_hh - The recurrent connections carry the previous output h⟨t-1⟩; it is not an instantaneous feedback - With many units, the state is a vector and the recurrent weights form a matrix: every unit's previous output feeds every unit - h⟨0⟩ = 0 means the network starts with an empty state
- A state is only useful if something reads it - Figure: the same diagram with an arrow labeled W_yh from the units up to the output y-hat at step t - This is the layer equation a = g(Wa + b) of the backpropagation lecture with one extra term, the recurrent input W_hh h⟨t-1⟩ - Compared with the running total: the state is a vector, the old state is mixed by a weight matrix instead of kept by lambda, and g keeps it from growing without bound - A leaky RNN combines both: h⟨t⟩ = (1 - Δt/τ) h⟨t-1⟩ + (Δt/τ) g(W_hh h⟨t-1⟩ + W_hx x⟨t⟩ + b_h). With Δt = τ it is the plain RNN above - For a character model, the readout at step t predicts the character at step t+1, as the transformer's readout did in the language-model lecture
- Figure (computed for this slide): three units, h⟨t⟩ = tanh(W_hh h⟨t−1⟩ + W_hx x⟨t⟩), weights set by hand, nothing trained. Each unit's connection onto itself is larger than 1, so the activity settles at a nonzero value and stays there - With recurrence, the units hold about 0.88, 0.54 and −0.86 from step 8 to step 40. Without it they are 0 from step 8 on - This is the model version of Funahashi's delay activity: a state that carries the cue after it is gone - The sign and size of each unit's held value depend on its input weight, so the three units together still carry what the cue was
- Step 1: the input x⟨1⟩ and the initial state h⟨0⟩ give h⟨1⟩, which gives the output ŷ⟨1⟩
- Step 2: the same weights again; h⟨1⟩ now plays the role h⟨0⟩ played
- To train it, draw the network as a chain of copies, one per time step - Figure: three copies of the same network, one per time step; magenta W_hh arrows carry each state to the next copy - Each copy is one time step, not a different network. Depth here is time: a 100-step sequence gives a 100-layer network - Because the weights are shared, a network trained on short sequences can run on longer ones. You met weight sharing in the convolutional-network lecture, where positions played the role that time steps play here - Contrast with the transformer: there, every position of a window is computed at once. Here step t cannot start before step t-1 has finished
- The unrolled network is an ordinary deep network, so ordinary backpropagation applies - The loss sums one cross-entropy term per step. Cross-entropy is -log of the probability the network gives to the correct next character, as in the language-model lecture; dividing by T gives the average, with the same best weights - The copy of the weights used at step k affects the loss at step k and at every later step; backpropagation through the unrolled chain collects all of that into one gradient - On very long sequences one uses truncated BPTT: unroll a fixed window, backpropagate within it, and carry the state forward to the next window - Werbos, P. J. (1990). Backpropagation through time: what it does and how to do it. Proceedings of the IEEE - Elman, J. L. (1990). Finding structure in time. Cognitive Science. He trained such a network to predict the next word; its hidden states grouped words by grammatical category without being told the categories
- The whole unit fits in a few lines of code - nn.Linear(a, b) holds a b-by-a weight matrix and, by default, a bias. W_hh has no bias of its own because b_h already sits in W_hx - The for loop is the unrolled network, and loss.backward() runs BPTT - This step-by-step computation is why a plain RNN trains slowly on a GPU: each pass waits for the one before. This is one reason transformers replaced RNNs (next lecture) - PyTorch also has nn.RNN and nn.GRU, which run the same computation in optimized code
- The gradient from far back is a long product, and that product can shrink or grow - w_hh is the leak λ of the leaky integrator, now written as what it is: a recurrent weight that training adjusts - Unrolling with no input: h⟨1⟩ = w_hh h⟨0⟩, h⟨2⟩ = w_hh h⟨1⟩ = w_hh² h⟨0⟩, …, h⟨T⟩ = w_hh^T h⟨0⟩ (a power, not a transpose: w_hh is one number). By the chain rule, ∂h⟨T⟩/∂h⟨0⟩ = w_hh · w_hh ⋯ w_hh = w_hh^T. A loss at step T sends its error back to step 0 multiplied by this factor; the gradient with respect to w_hh is a sum of such terms, one per step back - Figure (computed for this slide): w_hh^T against the number of steps back T, from 0 to 100, log axis. 1.02 → 7.2, 1.00 → 1, 0.98 → 0.13 at T = 100 - Why this hurts training: near 1, one gradient step can move the network from forgetting step 0 (0.13) to amplifying it (7.2). Below 1 the error signal from step 0 is too small to teach anything; above 1 it blows up - The derivative of h⟨T⟩ with respect to h⟨0⟩ is a product of one factor per step. For the one-unit linear recurrence every factor is w_hh - In a full RNN each factor is a matrix, the Jacobian diag(g'(z⟨t⟩)) W_hh, where z⟨t⟩ is the sum inside g at step t. Whether the product shrinks or grows depends on the size of W_hh (its largest singular value) and on the slopes g', which are at most 1 for tanh - Exploding gradients are the easy half: clip the gradient's length above a limit. Vanishing gradients are the hard half: clipping cannot create a gradient that is not there - Zucchet & Orvieto: even when the gradient neither vanishes nor explodes, the output's sensitivity to w_hh grows without bound as w_hh approaches 1, the "curse of memory" - Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks - Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. ICML - Zucchet, N., & Orvieto, A. (2024). Recurrent neural networks: vanishing and exploding gradients are not the end of the story. NeurIPS - Who found it: Sepp Hochreiter first analysed the vanishing gradient in his 1991 diploma thesis (Untersuchungen zu dynamischen neuronalen Netzen, TU Munich, advised by Jürgen Schmidhuber); Bengio, Simard & Frasconi (1994) reached the same result independently. The LSTM (Hochreiter & Schmidhuber 1997) was designed to fix it
- A fixed leak cannot both hold a cue and ignore distractors, so the unit is built up in three steps - Step 1 is the recurrent unit of the previous slides, now called the candidate: tanh of the weighted old state plus the weighted input
- Step 2: the update gate u is a second small network with a sigmoid output, one value between 0 and 1 per unit. The new state is a mix: keep 1 − u of the old state, write u of the candidate - Compare the leaky integrator, which keeps a fixed fraction λ. Here the kept fraction 1 − u differs for every unit and every step, and is learned - Why the old state appears twice: along the bottom it is an input to the gate and the candidate (it is read); along the top it is the memory itself (it is carried). Only the top path keeps it - Gate: u⟨t⟩ = σ(W_uh h⟨t−1⟩ + W_ux x⟨t⟩ + b_u). Papers call it z; here u, because z is the sum before the activation
- Step 3: the reset gate r decides how much of the old state goes into the candidate: candidate h̃⟨t⟩ = tanh(W_hh (r⟨t⟩ ⊙ h⟨t−1⟩) + W_hx x⟨t⟩ + b_h) - Figure: a thick green line carries the old state h⟨t-1⟩ from left to right through a multiply node (keep old, 1 - u) and an add node to the new state. Below, three violet boxes read the input and the old state: the update gate (sigma), the reset gate (sigma) and the candidate (tanh). The update gate sets both the keep node and the write node (u times the candidate). The reset gate feeds only the candidate - σ is the sigmoid, which squashes any number into 0 to 1; ⊙ multiplies entry by entry - Colors: violet = update gate u, blue = reset gate r, green = candidate h̃; the state h and the input x are black - Compare the leaky unit, which keeps a fixed fraction lambda. Here the kept fraction 1 - u⟨t⟩ differs for every unit and every step, and is learned - Gate: u⟨t⟩ = σ(W_uh h⟨t-1⟩ + W_ux x⟨t⟩ + b_u). Candidate: h̃⟨t⟩ = tanh(W_hh (r⟨t⟩ ⊙ h⟨t-1⟩) + W_hx x⟨t⟩ + b_h), with the reset gate r⟨t⟩ deciding how much of the old state enters it. Papers call the update gate z; here u, because z is the sum before the activation - The LSTM is the older gated unit, with three gates and a separate cell state; the GRU does about as well with fewer weights. The hGRU (next lecture, on Pathfinder) is a GRU laid out over the image - Cho, K., van Merriënboer, B., Gulcehre, C., et al. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. EMNLP - Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation
- The same unit, with its recurrent connections and shared weights, has been used wherever the input is a sequence, and also on static images, where the steps are steps of processing instead of moments of the input - Language: Elman's network predicted the next word and its hidden states grouped words by grammatical category. Mikolov et al. (2010) made recurrent language models practical for speech recognition - Translation: Sutskever, Vinyals & Le (2014): one recurrent network reads the sentence into a state, a second writes the translation from it - Speech: Graves, Mohamed & Hinton (2013): deep recurrent networks set the record on a standard speech benchmark (TIMIT) - Images: Vinyals et al. (2015) generate a caption word by word from an image. Spoerer, McClure & Kriegeskorte (2017) and Kubilius et al. (2019, CORnet) add recurrence to object-recognition networks and compare them with the ventral stream; our lab's hGRU comes later today - Cognitive science: Botvinick & Plaut (2004) model everyday routines (making coffee or tea) without a hierarchy of schemas, and reproduce the slips patients make - Neuroscience: Mante, Sussillo, Shenoy & Newsome (2013) and Yang et al. (2019) train networks on the monkey's task and compare their units with prefrontal neurons, as for the delay activity earlier
- Gated RNNs ran speech recognition and translation until about 2017. The vision lectures modeled the ventral stream as a feedforward network (V1, V2, V4, IT, one pass per image); this part asks where one pass stops explaining, and what adding recurrent circuits changed
- A behavioral test that gives feedback no time - Movie: eight trials of the rapid categorization experiment, four with an animal and four without, at four viewing distances. Each trial: fixation cross, a photograph for one video frame (about 17 ms), a blank (about 33 ms), then a noise mask (about 83 ms), a scrambled version of the same photograph - The mask follows the image onset by 50 ms and limits the time for feedback from higher areas to return. With the mask, most people still get most images right - Serre, Oliva & Poggio (2007): a feedforward model of simple and complex cell stages (HMAX) matched human observers from close-up heads to small, distant bodies. Performance is scored with d', a sensitivity measure from signal detection theory (hit rate against false-alarm rate; 0 is guessing) - Thorpe et al. (1996): with 20 ms flashes, the EEG separates animal from non-animal trials about 150 ms after the image - Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of processing in the human visual system. Nature - Serre, T., Oliva, A., & Poggio, T. (2007). A feedforward architecture accounts for rapid categorization. PNAS (Serre lab)
- A single sweep explains that case, so first draw the sweep on the anatomy - Figure: a macaque brain, lateral view (drawing by Retzius, 1906), with the ventral-stream areas V1, V2, V4 and IT marked and red arrows V1 to V2 to V4 to IT - The brain is shallow compared with a deep network: a handful of stages along the ventral stream. The sweep through them is what a flashed, masked image isolates - Kreiman, G., & Serre, T. (2020). Beyond the feedforward sweep: feedback computations in the visual cortex. Annals of the New York Academy of Sciences (Serre lab)
- Same brain, a second kind of connection added in blue - Horizontal connections link neurons within one area that look at different places in the image - Each blue arrow is a recurrent connection inside one area, drawn on V1, V2, V4 and IT
- Same brain, a third kind of connection added in green: from IT back to V4, V2 and V1 - Top-down connections run from higher areas to lower ones. By most counts, horizontal and top-down connections outnumber the feedforward ones - Recurrent processing: signals cycle through all three kinds of connection, so the response to one image keeps changing after the first sweep. This is depth in time, not in layers - Kreiman, G., & Serre, T. (2020). Beyond the feedforward sweep: feedback computations in the visual cortex. Annals of the New York Academy of Sciences. Brain drawing: Retzius (1906)
- The recurrent unit built earlier has no notion of space: W_hh is a full matrix, so every unit can reach every other in one step - Figure: cortical columns drawn as gray cylinders over a patch of visual field, with a red center, a green ring around it and a blue outer region. A cortical column is the set of neurons through the depth of cortex whose receptive fields sit at one location - In early visual cortex the columns are laid out as a map of the visual field (retinotopy), and horizontal connections link neurons a few millimeters apart, that is, neurons looking at nearby points - A convolution is exactly that constraint: the same small set of weights applied around every location. A recurrent layer with a convolutional W_hh is the hGRU of the coming slides
- A neuron in the central column receives a blue connection from a distant column, a green one from a neighboring column, and gold and red ones from neurons above and below it in its own column - In the model of Mély, Linsley & Serre (2018), near surround excites neurons with the same preferred orientation and far surround inhibits them; the rules were constrained by V1 recordings - Consequence: signals spread across the map step by step, so linking two distant points takes time that grows with their distance. This is the prediction tested by curve tracing (~49 ms per step) and by grouping reaction times - Top-down connections are the other kind of recurrence: higher areas see large parts of the image at once and send object-level information back to the map - Next lecture: curve tracing in V1, Pathfinder, grouping reaction times - Mély, D. A., Linsley, D., & Serre, T. (2018). Complementary surrounds explain diverse contextual phenomena across visual modalities. Psychological Review (Serre lab)
- With those connections, one circuit reproduces a family of illusions - Figure: three columns of contextual illusions. Tilt illusion: a center disc of vertical stripes inside a ring tilted slightly (top) or near horizontal (bottom). Color induction: an identical gray center on a pink ring (top) and a gray ring (bottom). Ring induction: concentric green, purple and orange rings around a white center, in two versions - The two centers in each column are physically identical; the surround changes what you see - Tilt illusion: repulsion (perceived tilt away from the surround) at small angle differences and attraction at large ones. The model reproduces this transition - The model's prediction that stimulating one excitatory neuron excites close neighbors and suppresses more distant, similarly tuned ones was later confirmed by single-neuron stimulation in mouse V1 (Chettih & Harvey 2019) - Next: the same circuit with learned gates, the hGRU, is trained on tasks that demand grouping - Mély, D. A., Linsley, D., & Serre, T. (2018). Complementary surrounds explain diverse contextual phenomena across visual modalities. Psychological Review (Serre lab)
- Each bullet maps to a part of the lecture: state and the two ways to use the past; working memory in cortex; the build-up from the integrator to the GRU; vanishing and exploding gradients; recurrence in visual cortex - Next lecture: why transformers replaced RNNs for language, how recurrence is coming back (hybrids, looped transformers), and world models: a recurrent state that carries the past can also be run forward to imagine the future
- Optional reading, for going deeper or as a starting point for the final project - Dao & Gu (2024): state-space models and attention as two forms of one family. Zucchet & Orvieto (2024): why long memories are hard to learn even when gradients behave. Feng et al. (2024): Were RNNs all we needed? Bick, Xing & Gu (2026): Retrieval-aware distillation for transformer-SSM hybrids - Kang, B., Midler, B., Chen, F., et al. (2026). Recurrent connections facilitate occluded object recognition by explaining-away. Nature Communications - Also cited today and worth reading: Linsley et al. (2018), Goetschalckx et al. (2023), Gu & Dao (2023) - More: Orvieto et al. (2023), Resurrecting recurrent neural networks for long sequences; Waleffe et al. (2024), An empirical study of Mamba-based language models; Rusch & Rus (2025), Oscillatory state-space models; Nayebi et al. (2022), Recurrent connections in the primate ventral visual stream
- Submit on Canvas, under Minute papers, as Minute Paper 19; the access code is read out in class - Credit is for a thoughtful attempt, not for being correct