The visual system & CNNs

Reader-friendly version of the 51 lecture slides: the lecture content in normal flow, with figure descriptions and data tables where the figure carries data. Open the presentation.

Slide 1

Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 11 · Theme 2: Learning representations: from neurons to deep networks

The visual system & CNNs

Tuesday, October 20 · Fall 2026

Return to contents

Slide 2

This week

Return to contents

Slide 3

How do we make a network generalize better?

Regularization: any change to training that gives up some fit to the training data to do better on new data

Return to contents

Slide 8

Regularization 1: a penalty on the weights

L(𝐰)  =  Ltrain(𝐰)⏟fit  +    λ∥𝐰∥2⏟strength    ∥𝐰∥2⏟size of the weights\mathcal{L}(\textcolor{#B5396B}{\mathbf{w}}) \;=\; \underbrace{\mathcal{L}_{\text{train}}(\textcolor{#B5396B}{\mathbf{w}})}_{\text{fit}} \;+\;\; \underbrace{\lambda\vphantom{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}}_{\text{strength}}\;\;\underbrace{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}_{\text{size of the weights}}

Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7

The same four panels with panel 4 highlighted and its fitted polynomial written out, y equals w 0 plus w 1 x up to w 9 x to the ninth, with the coefficients; the other panels dimmed. Four panels of the degree-9 polynomial fitted to the same 10 points, with an increasing penalty. No penalty: through every point, swinging far off, error on new points 0.86. Lambda 10 to the minus 7: a smooth curve close to the dashed one, 0.34. Lambda 10 to the minus 3: smooth, 0.34. Lambda 1: an almost flat line, training error 0.49, error on new points 0.61. Under each panel the size of the coefficient vector: 1.1 million, 189, 17, 0.65. Open green circles: 25 of the new points the error on new points is measured on

The same four panels, with the fourth panel, lambda equals 1, drawn in full and the others dimmed. Its fitted coefficients, written out below: y equals 0.35, minus 0.27 x, minus 0.34 x squared, minus 0.25 x cubed, minus 0.14 x to the fourth, minus 0.059 x to the fifth, plus 0.0053 x to the sixth, plus 0.053 x to the seventh, plus 0.088 x to the eighth, plus 0.11 x to the ninth. Under the panel: training RMSE 0.49, test RMSE 0.61, length of the weight vector 0.65. The point: every coefficient is under 0.35, the fitted curve is nearly a flat line, and the error rises on training and new points alike: too large a penalty underfits.

Open full-size figure

Return to contents

Slide 9

Ridge regression: the penalty on a neuron model

Two panels for one monkey IT recording site and 512 features of the trained network's late layer. Left: predicted against recorded responses for a ridge fit; training images faint magenta, held-out images green, both scattered along the diagonal, R-squared 0.71 on training and 0.69 on held-out images. Right: R-squared against the penalty lambda on a log axis; with almost no penalty, training R-squared is 0.85 and held-out R-squared 0.28; as the penalty grows, training R-squared falls while held-out R-squared rises to meet it near 0.69 at the penalty chosen on the training images, then both fall

Computed: one monkey IT recording site (Bao et al., 2020) predicted from 512 units of an ImageNet-trained ResNet-18's late layer by ridge regression, a random fifth of the 1,224 images held out. Left panel: predicted response (vertical, 0 to 60) against recorded response (horizontal, 0 to about 75), with a dashed diagonal where the two are equal; faint magenta dots are training images, larger green dots held-out images, both spread along the diagonal. Labels: held-out R squared 0.69, training R squared 0.71. Right panel: R squared (vertical, 0 to 1) against the penalty lambda on a log axis from 10 to the minus 2 to 10 to the 5. With almost no penalty the magenta training curve is at 0.85 and the green held-out curve at 0.28; as the penalty grows they meet near 0.69 at a dotted vertical line, the lambda chosen by cross-validation within the training images, then both fall. The point: too small a penalty overfits, and the penalty is chosen without the held-out images.

Open full-size figure

Computed: one monkey IT site (Bao et al. 2020), 1,224 object images, 512 late-layer units of ResNet-18; a random fifth held out

Return to contents

Slide 10

Regularization 2: dropout

Three copies of the same small network: in the first two, a different random half of the hidden units is switched off for one training step each, with their connections removed; in the third, at test time, all units are on

Schematic: three copies of the same small network, an input layer of four units, two hidden layers of five, and two output units, drawn as light blue circles joined by gray lines. Left, training step 1: five of the ten hidden units are crossed out and have no connections, one in the first hidden layer and four in the second. Middle, training step 2: a different five are crossed out, two in the first hidden layer and three in the second. Right, test: all units are on and every connection is drawn; caption, outputs scaled to match training, hidden units times 0.5. The point: at each training step a random half of the hidden units is switched off, so no unit can rely on a particular partner; at test time all are on.

Open full-size figure

Return to contents

Slide 15

Regularization 3: more examples

Computed: the same curve and noise, after Bishop (2006), Fig. 1.6

The same four panels with panel 4 highlighted and its fitted polynomial written out with the coefficients; the other panels dimmed. Four panels of the same degree-9 polynomial fitted to more and more noisy points from the dashed sine-shaped curve. N equals 10: the curve swings far off, error on new points 0.86. N equals 15: close to the dashed curve, 0.34. N equals 30: 0.35. N equals 100: the fitted curve follows the dashed one, training error 0.28, error on new points 0.31. Under each panel the size of the coefficient vector: 1.1 million, 46,000, 37,000, 8,700. Open green circles: 25 of the new points the error on new points is measured on

The same four panels, with the fourth panel, N equals 100 points, drawn in full and the others dimmed. Its fitted coefficients, written out below: y equals minus 0.24, plus 25 x, minus 260 x squared, plus 1.3 thousand x cubed, minus 3.4 thousand x to the fourth, plus 3.8 thousand x to the fifth, plus 450 x to the sixth, minus 5.1 thousand x to the seventh, plus 4.5 thousand x to the eighth, minus 1.3 thousand x to the ninth. Under the panel: training RMSE 0.28, test RMSE 0.31, length of the weight vector 8.7 thousand. The point: with 100 points the curve can no longer swing between them; the training error rises toward the noise level of 0.30 and the error on new points is the lowest of the four.

Open full-size figure

Return to contents

Slide 16

No more data? Make altered copies of it

Eight versions of one photograph of a pink orchid, each made by a torchvision transform: original, horizontal flip, resized crop, color jitter, rotation, affine warp, perspective, random erasing with a gray patch

Computed with torchvision on one THINGSplus photograph of a pink orchid in front of dark green leaves. Eight square versions in two rows of four, each labeled above. Top row: Original; Horizontal flip, the mirror image; Resized crop, a zoomed-in part of the flower filling the square; Color jitter, the same view with shifted color and brightness. Bottom row: Rotation, the photo turned by a few tens of degrees with black corners; Affine warp, sheared and shifted with black wedges at the edges; Perspective, the photo squeezed as if seen at an angle inside a black frame; Random erasing, the original with a gray rectangle covering part of the flower. The point: every version still shows an orchid, so each can be a new training example with the same label.

Open full-size figure

Return to contents

Slide 17

Augmentation in PyTorch: a transform pipeline

transforms.Compose([ transforms.RandomResizedCrop(224, scale=(0.5, 1.0)), # Crop and resize transforms.RandomHorizontalFlip(), # Flip horizontally at random transforms.RandomRotation(15), # Rotate by up to 15 degrees transforms.ColorJitter(0.3, 0.3, 0.3, 0.05), # Vary color and brightness transforms.ToTensor(), # Convert to channel-first floats in [0, 1] transforms.Normalize(IMAGENET_MEAN, IMAGENET_STD), # Standardize channels ])

Return to contents

Slide 18

From the multilayer perceptron to the convolutional network

A dense layer on an image needs a weight for every pixel, and an object can appear anywhere: a small filter, applied at every position

Return to contents

Slide 19

Dense versus convolutional: where the weights go

Two panels over the same 6 by 6 image of pixels. Left, dense: every pixel is wired to every one of six hidden units, a tangle of 216 thin lines, and every hidden unit to every output unit; caption: every pixel to every unit, 36 times 6 weights in the first layer, all different. Right, convolutional: a 4 by 4 map of hidden units, three of which are shown wired each to its own 3 by 3 patch of pixels with the same nine color-coded weights, and one second-layer unit wired to a 2 by 2 patch of the map; caption: each unit sees a 3 by 3 patch, the nine weights are shared by all sixteen units

Schematic: two panels over the same 6 by 6 gray-level image, a grid of gray squares at the bottom of each. Left, dense (multilayer perceptron): six hidden units above the image, each joined to all 36 pixels by thin gray lines, a tangle of 216 lines, and four output units joined to all six hidden units; caption, every pixel to every unit, 36 times 6 weights in the first layer, all different. Right, convolutional: a 4 by 4 map of hidden units; three are drawn wired each to its own 3 by 3 patch of pixels, the nine lines in the same nine colors for every unit; above, a 3 by 3 second-layer map with one unit wired to a 2 by 2 patch of the first map; caption, each unit sees a 3 by 3 patch, the 9 weights are shared by all 16 units. The point: a convolutional layer wires each unit to a local patch and reuses the same weights at every position.

Open full-size figure

Return to contents

Slide 20

Count the weights: a dense layer on an image

Horizontal bars on a log axis: 96 filters of 11 by 11 by 3, reused at every position, 34,848 weights; 96 dense units, each wired to every pixel of a 224 by 224 color image, 14,450,688 weights

Computed: two horizontal bars on a log axis labeled weights in the first layer (log scale), from 10 to the 3 to 10 to the 8. Top bar, magenta: 96 filters of 11 by 11 by 3, reused at every position, 34,848 weights. Bottom bar, slate gray and much longer: 96 dense units, each wired to every pixel of a 224 by 224 color image, 14,450,688 weights. The point: on a log axis the dense layer's bar runs almost three decades further, about 400 times more weights for the same 96 units.

Open full-size figure

Return to contents

Slide 21

Try it: template matching at every position

Return to contents

Slide 22

Try it: slide the filter yourself

Return to contents

Slide 23

Convolution: one filter, slid over the image

The tiger photograph in gray levels, 224 by 224, an arrow to one 11 by 11 vertical Gabor filter, an arrow to the feature map, 224 by 224, bright where the image has vertical stripes

Computed. Three items left to right joined by arrows. The tiger photograph in gray levels, labeled image 224 by 224. An 11 by 11 Gabor filter, labeled filter 11 by 11: a small grid of gray squares with a bright stripe between two dark stripes, tilted about 20 degrees from vertical. The feature map, labeled feature map 224 by 224: a dark square with bright magenta and orange streaks wherever the tiger has stripes at the filter's orientation, mostly on the body and legs. The point: one set of weights is applied at every position, and the map keeps the image's layout, so shifting the image shifts the map.

Open full-size figure

Return to contents

Slide 24

One filter gives one map; a layer gives a stack of maps

The same tiger at the same size, an arrow to four 11 by 11 Gabor filters at four orientations stacked as cards, an arrow to the four feature maps stacked as cards, all in the same color map as the single map before

Computed: the same tiger photograph, labeled image 224 by 224 by 1, an arrow to four 11 by 11 Gabor filters at four orientations stacked as offset cards, labeled 4 filters, 11 by 11, and an arrow to four feature maps stacked as offset cards in the same dark-to-orange color map as the single map before, labeled 4 feature maps: 224 by 224 by 4. The front map shows the near-vertical stripes; the maps behind it respond to the other three orientations. The point: each filter gives one map, so the layer's output is a stack with one channel per filter, and that stack is the next layer's input.

Open full-size figure

Computed: the tiger, four 11 × 11 Gabor filters, stride 1, padding 5, ReLU

Return to contents

Slide 25

The convolutional layer, in notation

𝐚[l]=g(𝐖[l]𝐚[l−1]+𝐛[l])\mathbf{a}^{[l]} = g(\mathbf{W}^{[l]}\mathbf{a}^{[l-1]} + \mathbf{b}^{[l]})

A[l]=g(K[l]∗A[l−1]+𝐛[l])\fmap{l} = \textcolor{#7A4FA3}{g}\big(\textcolor{#B5396B}{\kernel{l}} \conv \fmap{l-1} + \textcolor{#3F7A6B}{\mathbf{b}^{[l]}}\big)

Two panels over the same 6 by 6 image. Left, dense: every pixel wired to every hidden unit, a tangle of lines, the multilayer perceptron layer W a plus b. Right, convolutional: three hidden units of a 4 by 4 map each wired to its own 3 by 3 patch with the same nine color-coded weights, the convolution K star A plus b

Schematic: two panels over the same 6 by 6 gray-level image, a grid of gray squares at the bottom of each. Left, dense (multilayer perceptron): six hidden units above the image, each joined to all 36 pixels by thin gray lines, a tangle of 216 lines, and four output units joined to all six hidden units; caption, every pixel to every unit, 36 times 6 weights in the first layer, all different. Right, convolutional: a 4 by 4 map of hidden units; three are drawn wired each to its own 3 by 3 patch of pixels, the nine lines in the same nine colors for every unit; above, a 3 by 3 second-layer map with one unit wired to a 2 by 2 patch of the first map; caption, each unit sees a 3 by 3 patch, the 9 weights are shared by all 16 units. The point: a convolutional layer wires each unit to a local patch and reuses the same weights at every position.

Open full-size figure

Return to contents

Slide 26

Which brain areas turn an image into an object?

Simple and complex cells: the weighted sum and the pooling of the last slides, found in cortex decades earlier

Return to contents

Slide 27

Core object recognition: animal or not?

Return to contents

Slide 28

Monkeys do it too

Return to contents

Slide 29

The ventral stream: a hierarchy of areas

The ventral-stream hierarchy from retina and LGN through V1, V2, V4 to IT, with latencies growing from about 40 to 100 ms and neuron counts per stage

DiCarlo, Zoccolan and Rust (2012), Figure 1. Left: a side view of the macaque brain with the ventral visual stream colored, V1 in red at the back, V2 green, V4 blue, and the inferior temporal cortex in yellow divided into PIT, CIT and AIT toward the front; below, an eye, an arrow to a box labeled LGN, and an arrow from the LGN to V1. Right: the same areas stacked as boxes from bottom to top with the typical response latency beside each: LGN about 40 milliseconds, V1 about 50, V2 about 60, V4 about 70, PIT about 80, CIT about 90, AIT about 100. Each box carries an approximate neuron count, from about 190 million in V1 to about 16 million in AIT, and arrows between stages give the size of each representation, from about 1 million at the LGN to about 10 million at IT. The point: the ventral stream is a hierarchy of areas, each adding about 10 milliseconds.

Open full-size figure

Return to contents

Slide 30

Retina and LGN: Kuffler's center–surround cells

Stephen Kuffler

A black-and-white head-and-shoulders photograph of Stephen W. Kuffler, the physiologist who mapped the center-surround receptive fields of retinal ganglion cells in 1953. An older man with a high forehead and receding hair, smiling broadly, in a light open-collared shirt, looking slightly to the left of the camera. His name is printed under the photograph. The portrait sits in the top-right corner of the slide, above the caption Stephen Kuffler.

Open full-size figure
Stephen Kuffler
Two schematic receptive fields: an on-center cell, a disc of plus signs inside a ring of minus signs, and an off-center cell, the reverse; no labels

Schematic: two circular receptive fields side by side. Left, an on-center cell: a pink inner disc marked with three magenta plus signs, inside a light blue ring marked with four blue minus signs. Right, an off-center cell, the reverse: a light blue inner disc with minus signs inside a pink ring with plus signs. No labels. The point: light on the plus region raises the cell's firing and light on the minus region lowers it, so uniform light over both does almost nothing.

Open full-size figure
Kuffler 1953 as redrawn in Hubel's Eye, Brain and Vision: for an on-center retinal ganglion cell, four stimuli and their spike records: a small spot in the center gives a burst while the light is on; a larger spot covering center and surround gives little; a ring on the surround silences the cell and gives a burst when the light goes off; diffuse light over the whole field gives almost nothing

Kuffler's 1953 recordings as redrawn in Hubel's book Eye, Brain and Vision. Two panels, each a black rectangle with three white stimuli, a small spot, a larger spot and a ring, beside four rows of spike records; a bar under the records marks when the stimulus is on and off. Left panel, an on-center retinal ganglion cell: the top row is the resting discharge with no stimulus; the small spot on the center gives a dense burst while the light is on; the larger spot covering center and surround gives only a few spikes; the ring on the surround silences the cell while on and gives a burst at off. Right panel, an off-center cell, the mirror image: the small spot silences the cell and gives a burst at off, the large spot does little, the ring gives a burst while on. The point: these cells report local contrast, not brightness.

Open full-size figure

Return to contents

Slide 31

V1: Hubel & Wiesel's recording, on film

David Hubel and Torsten Wiesel at their recording setup

A black-and-white photograph of David Hubel and Torsten Wiesel in their laboratory, surrounded by racks of recording equipment, oscilloscopes and amplifiers. Hubel, wearing glasses and a dark sweater, sits on the left with one hand on the apparatus; Wiesel, in a patterned sweater, sits on a stool on the right. Both look toward the camera. The portrait sits in the top-right corner of the slide, above the caption Hubel and Wiesel.

Open full-size figure
Hubel & Wiesel

Return to contents

Slide 32

V1: Hubel & Wiesel's oriented cells

David Hubel and Torsten Wiesel at their recording setup

A black-and-white photograph of David Hubel and Torsten Wiesel in their laboratory, surrounded by racks of recording equipment, oscilloscopes and amplifiers. Hubel, wearing glasses and a dark sweater, sits on the left with one hand on the apparatus; Wiesel, in a patterned sweater, sits on a stool on the right. Both look toward the camera. The portrait sits in the top-right corner of the slide, above the caption Hubel and Wiesel.

Open full-size figure
Hubel & Wiesel
Hubel and Wiesel 1962, receptive fields of cat cortical cells mapped by hand with spots and bars of light: crosses mark positions where light excites the cell, triangles where it inhibits; panels a and b are circular center-surround maps, c to g are elongated maps with an ON stripe flanked by OFF regions at one orientation

Hubel and Wiesel (1962), Text-figure 2: seven receptive fields of cat cells mapped by hand with spots of light, labeled a to g. In the source's legend, crosses mark areas giving excitatory (on) responses and triangles areas giving inhibitory (off) responses; a continuous line through each elongated field marks its axis. Panels a and b, stacked at the left, are circular: a has crosses in the center inside a ring of triangles, on-center; b the reverse, off-center. Panels c to g are elongated along an oblique axis: c has a narrow stripe of crosses between flanks of triangles; d, e and f have a stripe of triangles between flanks of crosses; g has crosses on one side of the axis and triangles on the other, an edge. The point: cortical receptive fields are parallel excitatory and inhibitory strips at one orientation, not concentric rings.

Open full-size figure
The same seven receptive fields a to g redrawn with plus and minus regions: a, an on-center disc; b, an off-center disc; c, d, e and f, a plus stripe between minus flanks along the diagonal; g, an edge, plus on one side and minus on the other

Schematic redrawing of the seven hand maps a to g with pink plus regions and light blue minus regions. Panels a and b at the left are circles: a, a pink plus center inside a blue minus ring, on-center; b, a blue minus center inside a pink plus ring, off-center. Panels c, d, e and f are ellipses tilted about 48 degrees, each with a pink plus stripe along the long axis between two blue minus flanks. Panel g is an ellipse split along its long axis, blue minus on the upper-left half and pink plus on the lower-right half, an edge. The point: each oriented cell sums light with positive weights along a stripe and negative weights beside it.

Open full-size figure

Return to contents

Slide 33

V1 in one picture: ganglion → simple → complex

Three columns of icons: an on-center and an off-center circle; an oriented bar on a dark or light background at one position; an oriented bar with arrows showing it can move across the field

Three columns of black-and-white icons summarizing Hubel and Wiesel's V1. Left column, retinal ganglion or LGN cells: a black disc with a white spot in the center, an on-center cell, above a white disc with a black spot, an off-center cell. Middle column, simple cells: a black disc with a white oblique bar, above a white disc with a black oblique bar, each oriented and at one position. Right column, a complex cell: one large circle with a black oblique bar in the middle and a double-headed arrow across it, showing the bar can sit anywhere in the field at that orientation. The point: a weighted sum builds selectivity for orientation, and a pooling builds tolerance to position.

Open full-size figure

Return to contents

Slide 40

Mechanism: a row of LGN cells makes a simple cell

Hubel and Wiesel's wiring model, computed on model cells, bar at the right orientation but shifted 0.8 units sideways onto the inhibitory surrounds: four on-center LGN receptive fields in a row on a screen, each with a plus center and a minus surround, a yellow bar of light across them, four LGN cell bodies with spike trains below, all converging on one cortical simple cell with its own spike train

The same four model LGN cells converging on a cortical simple cell, with the bar at the row's orientation but shifted sideways by 0.8 units, now almost clear of the fields along their upper-left edge; the frame is labeled right orientation, shifted onto the surrounds. Spike trains: all four LGN cells and the simple cell show no spikes. The point: a simple cell is selective for position as well as orientation, because the bar must line up with the plus centers.

Open full-size figure

Return to contents

Slide 47

Mechanism: simple cells converge on a complex cell

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all three converging symmetrically on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

The same five model simple cells converging on a complex cell, with the bar past the right edge of field 5 at position 1.600. Spike trains: cell 5 three spikes, the others none, the complex cell three. Right panel: the dark complex-cell curve is now drawn over the whole range from minus 2 to 2: it rises from zero near minus 1.6, stays between about 0.98 and 1.1 across the five preferred positions with small dips between them, and falls back to zero past 1.6, while each gray simple-cell curve covers only a narrow range of positions. The point: each simple cell covers a narrow range of positions and the complex cell, their sum, a broad one, at the same orientation.

Open full-size figure

Return to contents

Slide 48

Recap

Return to contents

Slide 49

Further reading for the research trail

Optional: a few research-trail entries over the semester. Any paper mentioned in any lecture qualifies.

Return to contents

Slide 50

Further reading, continued

Return to contents

Slide 51

Minute paper

Submit on Canvas → Minute papers → Minute Paper 12 (access code read out in class).

Write three brief points in your own words:

  1. Hubel and Wiesel's simple and complex cells each correspond to one operation of a convolutional
    network. Name the two operations, and say what each buys the network.
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents