Carrying a state: RNNs & GRUs

Reader-friendly version of the 38 lecture slides: the lecture content in normal flow, with figure descriptions and data tables where the figure carries data. Open the presentation.

Slide 1

Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 18 · Theme 3: Dynamics, sequences & world models

Carrying a state: RNNs & GRUs

Thursday, November 19 · Fall 2026

Return to contents

Slide 2

Carrying a state

A transformer keeps every token it has read: its memory grows with the context. What does a brain keep?

Return to contents

Slide 3

Tasks that depend on the past

Three tasks in which the current item does not determine the answer. Language: the word bank means the edge of the water in "the river bank" and a financial institution in "the money in the bank". Vision: three frames over time, a ball to the left of a screen, the screen alone with nothing visible, the ball reappearing on the right. Arithmetic: 478 plus 256 equals 734, with carries of 1 above the tens and hundreds columns

Diagram drawn for this lecture, in three columns titled Language, Vision and Arithmetic. Language: the word bank is marked in red in two phrases, the river bank, which means the edge of the water, and the money in the bank, which means a financial institution. Vision: three frames in time order along an arrow labeled time. A blue ball sits to the left of a gray screen, then the screen stands alone with the label nothing visible, then the ball appears to the right of the screen. Arithmetic: 478 plus 256 equals 734, with a red carry of 1 above the tens and the hundreds columns. In each column the current item (a word, an empty frame, a pair of digits) does not fix the answer, and the earlier items do. The slide uses the three cases to show that a network needs the past in its input or in a state.

Open full-size figure

Return to contents

Slide 5

Using the past in a model: pass it in, or carry a state

Two panels with the same task: read the words the, cat, sat, on, the, and predict the next word, mat, in a box above the last word. Left, look back, attention: each earlier word keeps a stored key and value; the current word, the, sends a query whose arrows reach back to all four stored entries, labeled reads every earlier entry; the query leads up to the prediction. Stored: one entry per word, grows with the text. Right, carry forward, recurrence: a chain of states h1 to h5, each fed by its word and by the previous state, labeled updates one state, word by word; only the last state, h5, is solid and leads up to the prediction. Stored: one state h, fixed size

Diagram drawn for this lecture, in two panels over the same six words, the cat sat on the mat. Left panel, look back (attention): every earlier word keeps a stored key and value, and arrows from all five stored entries converge on the output for the current word, mat. The caption says one entry is stored per token and the store grows with T, the number of tokens. Right panel, carry forward (recurrence): a chain of states h1 to h6, each fed by its word and by the previous state. Only the current state h6 is drawn solid, and the caption says one state h of fixed size is stored. The slide contrasts the two ways of using the past: keep every earlier token and read them all, or keep one state that each new input updates.

Open full-size figure

Look back (attention): the past is passed in with the input → each new word, the whole sequence is reprocessed

Cost grows with the text: long chats must be compacted

Carry forward (recurrence): the past is in a state → one input at a time, fixed memory and work per step

Cost is fixed, but the past is compressed

Return to contents

Slide 6

Input vs. state: examples

Input vs. state: examples — table
Input 𝐱⟨t⟩\mathbf{x}^{\langle t\rangle} State 𝐡⟨t⟩\mathbf{h}^{\langle t\rangle}
Language the word bank what the text is about (river or money)
Vision a frame with the ball hidden where the ball is, how fast it moves
Arithmetic two digits the carry
Brain incoming spikes neural activity, voltages, synapses (slow: input adds, not resets)
Recurrent network a vector its units' activities, carried to the next step

Return to contents

Slide 7

A state is a lossy summary

Return to contents

Slide 8

A state in cortex: the delayed-response task

Reproduced from Funahashi, Bruce and Goldman-Rakic 1989, Fig. 1. A: timeline of the oculomotor delayed-response task: an intertrial interval, a fixation period F, a cue period C during which a peripheral cue is shown, a delay period D with only the fixation target, and a response period R followed by reward. B: one trial: the horizontal and vertical gaze traces stay flat from fixation to the end of the delay, then jump at the response; below, the spikes of one prefrontal neuron, dense through the whole delay

Published figure: Fig. 1 of Funahashi, Bruce and Goldman-Rakic (1989), the oculomotor delayed-response task in monkey prefrontal cortex. Panel A is a timeline of one trial: an intertrial interval, a fixation period F, a cue period C in which a peripheral cue is shown, a delay period D in which only the fixation target is on, and a response period R followed by reward. Panel B shows one trial: the horizontal and vertical gaze traces stay flat from fixation to the end of the delay and then jump at the response. Below the gaze traces are the spikes of one prefrontal neuron, which are dense through the whole delay. The slide gives the durations: the cue flashes for 0.5 s at one of 8 locations and the delay lasts 3 s. The gaze traces show that the monkey keeps its eyes on the fixation point during the delay, so the location of the cue must be held in memory.

Open full-size figure
  • Monkey prefrontal cortex (single-neuron recordings): the delayed-response task, a classic test of working memory
  • The monkey fixates (F); a cue (C) flashes at one of 8 locations; during a 3 s delay (D) nothing shows where it was; then it looks there (R)
  • Bottom trace, one trial: background firing in the intertrial interval (ITI), then dense firing through the delay

Return to contents

Slide 9

A state in cortex: delay activity

Reproduced from Funahashi, Bruce and Goldman-Rakic 1989, Fig. 3: the activity of one prefrontal neuron for cues at eight locations, arranged around a central diagram of the locations at 0, 45, 90, 135, 180, 225, 270 and 315 degrees. Each panel shows spike rasters over single trials and a histogram below, with vertical lines marking the cue C, the delay D and the response R. Only the panel for the cue at 270 degrees, at the bottom, shows dense firing through the whole 3-second delay; in the other panels the firing stays low or is suppressed

Published figure: Fig. 3 of Funahashi, Bruce and Goldman-Rakic (1989), the activity of one prefrontal neuron for cues at eight locations. A central diagram marks the fixation point and the eight cue locations at 0, 45, 90, 135, 180, 225, 270 and 315 degrees, and the panel for each location is placed around it. Each panel has spike rasters for single trials, one row per trial, and a histogram of the average firing below. Vertical lines mark the cue C, the delay D and the response R; the scale bars are 50 spikes per second and 1 s. Only the panel for the cue at 270 degrees, at the bottom, shows dense firing through the whole 3-second delay. In the other panels the firing stays low or is suppressed. The slide calls 270 degrees the preferred location of this neuron and uses the figure to show that the cued location is held in activity after the cue is gone.

Open full-size figure
  • One neuron, 8 cue locations (center); each panel: single trials, average below
  • It fires through the delay only when the cue was at the bottom (270°): its preferred location. Cue at the top (45°–135°): below its ITI rate
  • So the neuron is selective, and it keeps its selectivity through the delay: a population of such neurons holds the cued location
  • Model: a recurrent network trained on this task develops units with the same delay activity → next, we build that recurrent unit

Return to contents

Slide 10

Building a recurrent unit

Goal: extend the model neuron of earlier lectures so it can hold a state

Return to contents

Slide 11

A first state: an integrator

Computed here, over time steps 1 to 40. Top: the input x, blue stems of plus 1 at steps 5, 6, 7 and 15 and minus 1 at steps 25 and 26, zero elsewhere. Bottom: the running total h, a step curve that climbs to 3 after the first three pulses, to 4 after step 15, drops to 2 after the two negative pulses and stays at 2 until step 40

Computed here, over time steps 1 to 40. Top: the input x, blue stems of plus 1 at steps 5, 6, 7 and 15 and minus 1 at steps 25 and 26, zero elsewhere. Bottom: the running total h, a step curve that climbs to 3 after the first three pulses, to 4 after step 15, drops to 2 after the two negative pulses and stays at 2 until step 40

Open full-size figure
  • Start from the linear model neuron, output =w x=\textcolor{#B5396B}{w}\,x, with one change: time is an explicit index tt, one input x⟨t⟩x^{\langle t\rangle} per step
  • Add the old state:   h⟨t⟩=h⟨t−1⟩+w x⟨t⟩\;h^{\langle t\rangle}=h^{\langle t-1\rangle}+\textcolor{#B5396B}{w}\,x^{\langle t\rangle} (an integrator; here w=1\textcolor{#B5396B}{w}=1)
  • The state sums all inputs so far, and holds the sum once the input stops

Return to contents

Slide 12

A leaky integrator

The same input and running total, with a second, green curve: the leaky state with lambda 0.8. It rises to 2.44 at step 7 and decays toward zero between pulses (0.46 at step 20), dips below zero after the negative pulses (minus 0.69 at step 30) and is back near zero, minus 0.07, at step 40, while the running total still holds 2

The same input and running total, with a second, green curve: the leaky state with lambda 0.8. It rises to 2.44 at step 7 and decays toward zero between pulses (0.46 at step 20), dips below zero after the negative pulses (minus 0.69 at step 30) and is back near zero, minus 0.07, at step 40, while the running total still holds 2

Open full-size figure
  • Not plausible for a neuron: its voltage leaks back to rest (time constant τ\tau ≈ 10–20 ms)
  • Leaky integrator: keep a fraction λ\textcolor{#B5396B}{\lambda} (between 0 and 1):   h⟨t⟩=λ h⟨t−1⟩+w x⟨t⟩\;h^{\langle t\rangle}=\textcolor{#B5396B}{\lambda}\,h^{\langle t-1\rangle}+\textcolor{#B5396B}{w}\,x^{\langle t\rangle}
  • λ=1\textcolor{#B5396B}{\lambda}=1 is the integrator: nothing is lost. λ<1\textcolor{#B5396B}{\lambda}<1: older inputs fade

Return to contents

Slide 13

From one unit to a network: the input

The recurrent unit, step 1 of 4: the input vector x at step t, drawn as four bars, sends an arrow labeled W h x up into a box of four units

Diagram drawn for this lecture, the first of four frames that build one recurrent unit. The input vector x at step t is drawn as four bars at the bottom. An arrow labeled W h x leads up from it into a box of four units. The frame adds the input and its weights, and nothing else: the units compute a weighted sum of the input. The slide reads subscripts as destination from source, so W h x goes to the units h from the input x.

Open full-size figure

𝐖hx 𝐱⟨t⟩\textcolor{#B5396B}{\mathbf{W}_{hx}}\,\textcolor{#3E6D8E}{\mathbf{x}^{\langle t\rangle}}

  • Input 𝐱⟨t⟩\mathbf{x}^{\langle t\rangle}: a vector at step tt (a word, an image)
  • 𝐖hx\textcolor{#B5396B}{\mathbf{W}_{hx}}: weights from the input to the units (subscripts read destination ← source)

Return to contents

Slide 14

From one unit to a network: the state

The recurrent unit, step 2 of 4: the same diagram; the units now carry the label h at step t, their activities, and a small g marks the nonlinearity

Diagram drawn for this lecture, the second of four frames building the recurrent unit. The input and the weights W h x are as in the first frame. The four units now carry the label h at step t, their activities, and a small g marks the nonlinearity applied to the weighted sum, usually tanh, with a bias. The frame adds the state and the nonlinearity. At this point the layer is feedforward, since the same input always gives the same output.

Open full-size figure

𝐡⟨t⟩=g(𝐖hx 𝐱⟨t⟩+𝐛h)\textcolor{#3F7A6B}{\mathbf{h}^{\langle t\rangle}}=\textcolor{#7A4FA3}{g}\big(\textcolor{#B5396B}{\mathbf{W}_{hx}}\,\textcolor{#3E6D8E}{\mathbf{x}^{\langle t\rangle}}+\mathbf{b}_h\big)

  • The units' activities form the state 𝐡⟨t⟩\mathbf{h}^{\langle t\rangle}
  • gg: the nonlinearity (usually tanh⁡\tanh); 𝐛h\mathbf{b}_h: a bias
  • So far, a feedforward layer: the output depends on the current input only

Return to contents

Slide 15

From one unit to a network: recurrent connections

The recurrent unit, step 3 of 4: the same diagram with recurrent connections added in green, labeled W h h and h at step t minus 1, last step's activities. Every unit has a loop onto itself; for unit h1, one curved arrow from each of h2, h3 and hD shows the connections between units

Diagram drawn for this lecture, the third of four frames building the recurrent unit. The frame adds a loop. An arrow leaves the units on the right, labeled h at step t minus 1, and comes back into them on the left through the weights W h h. The state at step t now depends on the input and on the previous state, which makes the unit a recurrent neural network. The loop carries the previous activities, not an instantaneous feedback, and the state starts at zero.

Open full-size figure

𝐡⟨t⟩=g(𝐖hh 𝐡⟨t−1⟩+𝐖hx 𝐱⟨t⟩+𝐛h)\textcolor{#3F7A6B}{\mathbf{h}^{\langle t\rangle}}=\textcolor{#7A4FA3}{g}\big(\textcolor{#3F7A6B}{\mathbf{W}_{hh}}\,\mathbf{h}^{\langle t-1\rangle}+\textcolor{#B5396B}{\mathbf{W}_{hx}}\,\textcolor{#3E6D8E}{\mathbf{x}^{\langle t\rangle}}+\mathbf{b}_h\big)

  • One addition: the units' previous activities 𝐡⟨t−1⟩\mathbf{h}^{\langle t-1\rangle} come back in, through 𝐖hh\textcolor{#3F7A6B}{\mathbf{W}_{hh}}
  • The state now depends on the input and on the previous state: a recurrent neural network (RNN), often called a vanilla RNN (or Elman network)
  • Start: 𝐡⟨0⟩=𝟎\mathbf{h}^{\langle 0\rangle}=\mathbf{0}

Return to contents

Slide 16

From one unit to a network: the readout

The recurrent unit, step 4 of 4: the same diagram with a readout added: an arrow labeled W y h goes up from the units to the output y-hat at step t

Diagram drawn for this lecture, the fourth and last frame building the recurrent unit. The frame adds a readout. An arrow labeled W y h goes up from the units to the output y-hat at step t. The readout maps the state to one score per output, and a softmax turns the scores into probabilities, as in the output layer of a language model. The three sets of weights, W h x, W h h and W y h, are the same at every step.

Open full-size figure

𝐡⟨t⟩=g(𝐖hh 𝐡⟨t−1⟩+𝐖hx 𝐱⟨t⟩+𝐛h)\textcolor{#3F7A6B}{\mathbf{h}^{\langle t\rangle}}=\textcolor{#7A4FA3}{g}\big(\textcolor{#3F7A6B}{\mathbf{W}_{hh}}\,\mathbf{h}^{\langle t-1\rangle}+\textcolor{#B5396B}{\mathbf{W}_{hx}}\,\textcolor{#3E6D8E}{\mathbf{x}^{\langle t\rangle}}+\mathbf{b}_h\big)

𝐲^⟨t⟩=softmax(𝐖yh 𝐡⟨t⟩+𝐛y)\hat{\mathbf{y}}^{\langle t\rangle}=\mathrm{softmax}\big(\textcolor{#2C5282}{\mathbf{W}_{yh}}\,\textcolor{#3F7A6B}{\mathbf{h}^{\langle t\rangle}}+\mathbf{b}_y\big)

  • Readout: 𝐖yh\textcolor{#2C5282}{\mathbf{W}_{yh}} maps the state to one score per output (the logits)
  • Softmax → probabilities, as in a language model's output layer
  • Three sets of weights, the same at every step

Return to contents

Slide 17

The state over time: activity outlasts the input

Computed for this slide: three hidden units of a small recurrent network over 40 time steps. Top: the input, a cue of 1 at steps 5, 6 and 7, then nothing (the delay). Middle, with the loop: the three units jump at the cue and stay there for the rest of the 40 steps, near 0.88, 0.54 and minus 0.86. Bottom, without the loop (W h h set to 0): the same units follow the cue and fall back to 0 at step 8

Computed for this slide: three hidden units of a small recurrent network over 40 time steps. Top: the input, a cue of 1 at steps 5, 6 and 7, then nothing (the delay). Middle, with the loop: the three units jump at the cue and stay there for the rest of the 40 steps, near 0.88, 0.54 and minus 0.86. Bottom, without the loop (W h h set to 0): the same units follow the cue and fall back to 0 at step 8

Open full-size figure
  • The vanilla RNN of the previous slides, 3 hidden units, weights set by hand (not trained), run for 40 steps
  • A cue at t=5t=5–77, then no input, as in the delayed-response task
  • With recurrence, activity outlasts the cue and is held; without it (𝐖hh=𝟎\textcolor{#3F7A6B}{\mathbf{W}_{hh}}=\mathbf{0}), it stops with the input

Return to contents

Slide 20

To train a recurrent network, unroll it in time

The recurrent network unrolled over three time steps, t = 1, 2, 3. At each step, an input column x1 to xD at the bottom feeds, through magenta weights W_hx, a column of hidden units h1 to hD, which feeds an output column y-hat 1 and 2 through W_yh. Magenta arrows W_hh link each hidden column to the next one, starting from h at step 0 on the left. Caption: same W_hx, W_hh, W_yh at every step

The recurrent network unrolled over three time steps, t = 1, 2, 3. At each step, an input column x1 to xD at the bottom feeds, through magenta weights W_hx, a column of hidden units h1 to hD, which feeds an output column y-hat 1 and 2 through W_yh. Magenta arrows W_hh link each hidden column to the next one, starting from h at step 0 on the left. Caption: same W_hx, W_hh, W_yh at every step

Open full-size figure

Return to contents

Slide 21

Training: backprop through time

L=∑t=1Tℓ(𝐲^⟨t⟩, 𝐲⟨t⟩)\mathcal{L}=\sum_{t=1}^{T}\ell\big(\hat{\mathbf{y}}^{\langle t\rangle},\,\mathbf{y}^{\langle t\rangle}\big)

Return to contents

Slide 22

Code: an RNN in PyTorch

import torch, torch.nn as nn class RNN(nn.Module): def __init__(self, d_in, d_hidden): super().__init__() self.W_hx = nn.Linear(d_in, d_hidden) # x -> h (+ b_h) self.W_hh = nn.Linear(d_hidden, d_hidden, bias=False) # h -> h def forward(self, x_seq): # x_seq: (T, d_in) h = torch.zeros(self.W_hh.out_features) H = [] for x_t in x_seq: # unroll over time h = torch.tanh(self.W_hh(h) + self.W_hx(x_t)) # h^{<t>} H.append(h) return torch.stack(H) # (T, d_hidden)

The for loop over time is the unrolled network; loss.backward() runs BPTT.

Return to contents

Slide 23

Why long memories are hard to learn

  • Leaky unit, its leak now a learned recurrent weight: h⟨t⟩=whh h⟨t−1⟩+w x⟨t⟩h^{\langle t\rangle}=\textcolor{#3F7A6B}{w_{hh}}\,h^{\langle t-1\rangle}+w\,x^{\langle t\rangle}
  • No input after the first step: h⟨T⟩=(whh)T h⟨0⟩h^{\langle T\rangle}=(\textcolor{#3F7A6B}{w_{hh}})^{T}\,h^{\langle 0\rangle}
  • Gradient reaching the first step: ∂h⟨T⟩∂h⟨0⟩=(whh)T\dfrac{\partial h^{\langle T\rangle}}{\partial h^{\langle 0\rangle}}=(\textcolor{#3F7A6B}{w_{hh}})^{T}

whh<1\textcolor{#3F7A6B}{w_{hh}}<1: it vanishes · whh>1\textcolor{#3F7A6B}{w_{hh}}>1: it explodes

  • After 100 steps, the gradient factor is 0.13 for whh=0.98\textcolor{#3F7A6B}{w_{hh}}=0.98 and 7.2 for 1.021.02 (plot): a 50× change, so gradient steps overshoot
Computed for this slide: w_hh to the power T against the number of steps back T, 0 to 100, log axis from 0.05 to 20. With w_hh = 1.02 the curve rises to 7.2 at T = 100; with 1.00 it stays at 1; with 0.98 it falls to 0.13

Computed for this slide: w_hh to the power T against the number of steps back T, 0 to 100, log axis from 0.05 to 20. With w_hh = 1.02 the curve rises to 7.2 at T = 100; with 1.00 it stays at 1; with 0.98 it falls to 0.13

Open full-size figure

Return to contents

Slide 24

Gating, step 1: the vanilla RNN

Gating, step 1 of 3: the plain recurrent unit. The input x at t and the old state h at t minus 1 run along two black lines at the bottom and feed a green box, the candidate h tilde (tanh); its output goes straight up to the new state h at t

Gating, step 1 of 3: the plain recurrent unit. The input x at t and the old state h at t minus 1 run along two black lines at the bottom and feed a green box, the candidate h tilde (tanh); its output goes straight up to the new state h at t

Open full-size figure

𝐡⟨t⟩=tanh⁡(𝐖hh𝐡⟨t−1⟩+𝐖hx𝐱⟨t⟩)\mathbf{h}^{\langle t\rangle}=\tanh(\textcolor{#3F7A6B}{\mathbf{W}_{hh}}\mathbf{h}^{\langle t-1\rangle}+\textcolor{#B5396B}{\mathbf{W}_{hx}}\mathbf{x}^{\langle t\rangle})

  • The vanilla RNN: every step rewrites the whole state, so information from far back fades (previous slide)
  • A gated unit (next) calls this result the candidate 𝐡~⟨t⟩\textcolor{#3F7A6B}{\tilde{\mathbf{h}}^{\langle t\rangle}} and decides how much of it to keep; in the vanilla RNN, 𝐡⟨t⟩=𝐡~⟨t⟩\mathbf{h}^{\langle t\rangle}=\textcolor{#3F7A6B}{\tilde{\mathbf{h}}^{\langle t\rangle}}

Return to contents

Slide 25

Gating, step 2: an update gate

Gating, step 2 of 3: the same diagram with the update gate added. A violet box, update u (sigma), reads x and the old state. The old state now also runs along the top, through a multiply node labeled keep old, 1 minus u, and an add node, to the new state. The candidate is multiplied by u (write new) before it is added

Gating, step 2 of 3: the same diagram with the update gate added. A violet box, update u (sigma), reads x and the old state. The old state now also runs along the top, through a multiply node labeled keep old, 1 minus u, and an add node, to the new state. The candidate is multiplied by u (write new) before it is added

Open full-size figure

𝐡~⟨t⟩=tanh⁡(𝐖hh𝐡⟨t−1⟩+𝐖hx𝐱⟨t⟩)𝐡⟨t⟩=(1−𝐮⟨t⟩)⊙𝐡⟨t−1⟩+𝐮⟨t⟩⊙𝐡~⟨t⟩\begin{gathered}\textcolor{#3F7A6B}{\tilde{\mathbf{h}}^{\langle t\rangle}}=\tanh(\textcolor{#3F7A6B}{\mathbf{W}_{hh}}\mathbf{h}^{\langle t-1\rangle}+\textcolor{#B5396B}{\mathbf{W}_{hx}}\mathbf{x}^{\langle t\rangle})\\ \mathbf{h}^{\langle t\rangle}=(1-\textcolor{#7A4FA3}{\mathbf{u}^{\langle t\rangle}})\odot\mathbf{h}^{\langle t-1\rangle}+\textcolor{#7A4FA3}{\mathbf{u}^{\langle t\rangle}}\odot\textcolor{#3F7A6B}{\tilde{\mathbf{h}}^{\langle t\rangle}}\end{gathered}

  • Update gate 𝐮⟨t⟩=σ(⋅)\textcolor{#7A4FA3}{\mathbf{u}^{\langle t\rangle}}=\sigma(\cdot), between 0 and 1, from the input and the state: an input-dependent leak
  • The old state is used twice: read (bottom) to compute the gate and the candidate; carried (top), scaled by 1−𝐮1-\textcolor{#7A4FA3}{\mathbf{u}}
  • In plain words: each unit learns when to keep its old value and when to overwrite it; in practice gradients survive many more steps

Return to contents

Slide 26

Gating, step 3: a reset gate → the GRU

Gating, step 3 of 3, the full GRU: a blue box, reset r (sigma), is added between the update gate and the candidate; it reads x and the old state and feeds the candidate

Gating, step 3 of 3, the full GRU: a blue box, reset r (sigma), is added between the update gate and the candidate; it reads x and the old state and feeds the candidate

Open full-size figure

𝐡~⟨t⟩=tanh⁡(𝐖hh(𝐫⟨t⟩⊙𝐡⟨t−1⟩)+𝐖hx𝐱⟨t⟩)𝐡⟨t⟩=(1−𝐮⟨t⟩)⊙𝐡⟨t−1⟩+𝐮⟨t⟩⊙𝐡~⟨t⟩\begin{gathered}\textcolor{#3F7A6B}{\tilde{\mathbf{h}}^{\langle t\rangle}}=\tanh(\textcolor{#3F7A6B}{\mathbf{W}_{hh}}(\textcolor{#3E6D8E}{\mathbf{r}^{\langle t\rangle}}\odot\mathbf{h}^{\langle t-1\rangle})+\textcolor{#B5396B}{\mathbf{W}_{hx}}\mathbf{x}^{\langle t\rangle})\\ \mathbf{h}^{\langle t\rangle}=(1-\textcolor{#7A4FA3}{\mathbf{u}^{\langle t\rangle}})\odot\mathbf{h}^{\langle t-1\rangle}+\textcolor{#7A4FA3}{\mathbf{u}^{\langle t\rangle}}\odot\textcolor{#3F7A6B}{\tilde{\mathbf{h}}^{\langle t\rangle}}\end{gathered}

  • Reset gate 𝐫⟨t⟩\textcolor{#3E6D8E}{\mathbf{r}^{\langle t\rangle}}: how much old state enters the candidate
  • Update and reset gates together: the GRU (gated recurrent unit), a simplified LSTM (long short-term memory: three gates and a separate memory cell)

Return to contents

Slide 27

Where recurrent networks have been used

Where recurrent networks have been used — table
Domain What the network does Examples
Language predicts the next word Elman 1990; Mikolov 2010
Translation reads a sentence, writes a translation Sutskever 2014
Speech turns sound into text Graves 2013
Images captions; object recognition in steps Vinyals 2015; Spoerer 2017; Kubilius 2019
Cognition everyday routines and their slips Botvinick & Plaut 2004
Neuroscience does a monkey's task; units vs. neurons Mante 2013; Yang 2019

Return to contents

Slide 28

Recurrence in visual cortex

One image, held fixed: the state changes over processing steps and links distant parts of the image.

Return to contents

Slide 29

A feedforward sweep recognizes a flashed image

Animation of the masked rapid-categorization task, looping over eight trials: a blank screen, a fixation cross, a grayscale photograph flashed for 20 ms, a short blank, then a noise mask made from the same photograph; four photographs contain an animal and four do not

Animation of the masked rapid-categorization task, looping over eight trials: a blank screen, a fixation cross, a grayscale photograph flashed for 20 ms, a short blank, then a noise mask made from the same photograph; four photographs contain an animal and four do not

Open full-size figure
  • Human behavior. Rapid categorization: animal or not, from a single flash
  • Image for 20 ms, then a mask 50 ms after onset → little time for recurrent processing
  • With an early mask, one feedforward sweep is enough

Return to contents

Slide 32

Cortex is not a single sweep

V1V2V4IT

  • Monkey visual cortex (anatomy): ventral stream V1 → V2 → V4 → IT
  • Feedforward sweep: one pass up the hierarchy; what HMAX and CNNs model
  • Horizontal connections within each area
  • Top-down (feedback) connections from higher to lower areas
  • Depth in time, not in layers; needed when parts of the image must be linked across space

Return to contents

Slide 33

From the recurrent unit to visual cortex

Schematic of cortical columns drawn as gray cylinders above a patch of visual field: a red center, a green ring around it and a blue outer region. A neuron in the central column receives a blue connection from a neuron in a distant column, a green connection from a neuron in a neighboring column, and gold and red connections from neurons above and below it in its own column

Diagram from an earlier lecture of this course, showing cortical columns as gray cylinders above a patch of visual field. The patch has a red center, a green ring around it and a blue outer region. A neuron in the central column receives a blue connection from a neuron in a distant column, a green connection from a neuron in a neighboring column, and gold and red connections from neurons above and below it in its own column. The drawing follows the circuit model of Mély, Linsley and Serre (2018). The red region is the classical receptive field, which drives the neuron. The green and blue regions are the surround, which changes the response through horizontal connections: the near surround excites neurons with the same preferred orientation and the far surround inhibits them.

Open full-size figure
  • In the RNN so far, 𝐖hh\textcolor{#3F7A6B}{\mathbf{W}_{hh}} connects every unit to all others: no positions in space
  • Visual cortex is retinotopic: a cortical column (gray) sees one location; neighboring columns see neighboring locations
  • Horizontal connections are local and the same everywhere (𝐖hh\textcolor{#3F7A6B}{\mathbf{W}_{hh}} becomes a convolution)

Return to contents

Slide 34

Horizontal connections link distant parts of the image

Schematic of cortical columns drawn as gray cylinders above a patch of visual field: a red center, a green ring around it and a blue outer region. A neuron in the central column receives a blue connection from a neuron in a distant column, a green connection from a neuron in a neighboring column, and gold and red connections from neurons above and below it in its own column

Diagram from an earlier lecture of this course, showing cortical columns as gray cylinders above a patch of visual field. The patch has a red center, a green ring around it and a blue outer region. A neuron in the central column receives a blue connection from a neuron in a distant column, a green connection from a neuron in a neighboring column, and gold and red connections from neurons above and below it in its own column. The drawing follows the circuit model of Mély, Linsley and Serre (2018). The red region is the classical receptive field, which drives the neuron. The green and blue regions are the surround, which changes the response through horizontal connections: the near surround excites neurons with the same preferred orientation and the far surround inhibits them.

Open full-size figure
  • In the RNN so far, 𝐖hh\textcolor{#3F7A6B}{\mathbf{W}_{hh}} connects every unit to all others: no positions in space
  • Visual cortex is retinotopic: a cortical column (gray) sees one location; neighboring columns see neighboring locations
  • Horizontal connections are local and the same everywhere (𝐖hh\textcolor{#3F7A6B}{\mathbf{W}_{hh}} becomes a convolution)
  • Center (red) drives the neuron; the surround (green, blue) modulates it
  • Activity spreads across the map one neighborhood per step: linking distant parts takes many steps

Return to contents

Slide 35

One circuit explains contextual illusions

Contextual illusions. Left column, tilt illusion: a central disc of vertical black and white stripes inside a ring of stripes tilted slightly from vertical in the top row and close to horizontal in the bottom row. Middle column, color induction: an identical gray center on a pink ring in the top row and on a gray ring in the bottom row. Right column, ring induction: concentric rings of green, purple and orange around a white center, in two versions

Figure of three contextual illusions, one per column, each with a top and a bottom version. Left, tilt illusion: a central disc of vertical black and white stripes inside a ring of stripes tilted slightly from vertical in the top row and close to horizontal in the bottom row. Middle, color induction: an identical gray center on a pink ring in the top row and on a gray ring in the bottom row. Right, ring induction: concentric rings of green, purple and orange around a white center, in two versions. The two centers in each column are physically identical, and the surround changes what they look like. The slide credits these illusions to the circuit model of Mély, Linsley and Serre (2018), which reproduces them with one set of parameters.

Open full-size figure

The two centers in each column are physically identical

  • Human perception. Tilt illusion: a tilted surround shifts the perceived orientation of the center
  • Color induction: the same gray center looks different on a pink and on a gray surround
  • Model: with one set of parameters, near excitation and far inhibition reproduce these shifts in orientation, color, motion and depth
  • If illusions are side effects of linking fragments into objects, tasks that require grouping should need this circuit

Return to contents

Slide 36

Recap

Return to contents

Slide 37

Further reading

Optional: for going deeper, or as a starting point for the final project.

Return to contents

Slide 38

Minute paper

Submit on Canvas → Minute papers → Minute Paper 19 (access code read out in class).

Write three brief points in your own words:

  1. Visual cortex is full of local horizontal connections. Why would a network built this way take longer to group the parts of some images than others? What in today's lecture (neurons, behavior, models) supports this?
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents