Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 11 · Theme 2: Learning representations: from neurons to deep networks

The visual system & CNNs

Tuesday, October 20 · Fall 2026

Reader-friendly version — lecture content with figure descriptions

This week

  • Exam 1: Thursday, in class. It covers the first eight lectures and the four steps of backpropagation
  • Recitation tonight, 6:30 PM: exam preparation
  • The third assignment is released Thursday

How do we make a network generalize better?

Regularization: any change to training that gives up some fit to the training data to do better on new data

Regularization 1: a penalty on the weights

L(w)  =  Ltrain(w)⏟fit  +    λ∥w∥2⏟strength    ∥w∥2⏟size of the weights\mathcal{L}(\textcolor{#B5396B}{\mathbf{w}}) \;=\; \underbrace{\mathcal{L}_{\text{train}}(\textcolor{#B5396B}{\mathbf{w}})}_{\text{fit}} \;+\;\; \underbrace{\lambda\vphantom{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}}_{\text{strength}}\;\;\underbrace{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}_{\text{size of the weights}}

Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7

Four panels of the degree-9 polynomial fitted to the same 10 points, with an increasing penalty. No penalty: through every point, swinging far off, error on new points 0.86. Lambda 10 to the minus 7: a smooth curve close to the dashed one, 0.34. Lambda 10 to the minus 3: smooth, 0.34. Lambda 1: an almost flat line, training error 0.49, error on new points 0.61. Under each panel the size of the coefficient vector: 1.1 million, 189, 17, 0.65. Open green circles: 25 of the new points the error on new points is measured on

  • Weight penalty = weight decay = L2; on least squares, ridge regression. ∥w∥\lVert\mathbf{w}\rVert drops from a million to 0.65; too large a λ\lambda underfits

Regularization 1: a penalty on the weights

L(w)  =  Ltrain(w)⏟fit  +    λ∥w∥2⏟strength    ∥w∥2⏟size of the weights\mathcal{L}(\textcolor{#B5396B}{\mathbf{w}}) \;=\; \underbrace{\mathcal{L}_{\text{train}}(\textcolor{#B5396B}{\mathbf{w}})}_{\text{fit}} \;+\;\; \underbrace{\lambda\vphantom{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}}_{\text{strength}}\;\;\underbrace{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}_{\text{size of the weights}}

Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7

The same four panels with panel 1 highlighted and its fitted polynomial written out, y equals w 0 plus w 1 x up to w 9 x to the ninth, with the coefficients; the other panels dimmed. Four panels of the degree-9 polynomial fitted to the same 10 points, with an increasing penalty. No penalty: through every point, swinging far off, error on new points 0.86. Lambda 10 to the minus 7: a smooth curve close to the dashed one, 0.34. Lambda 10 to the minus 3: smooth, 0.34. Lambda 1: an almost flat line, training error 0.49, error on new points 0.61. Under each panel the size of the coefficient vector: 1.1 million, 189, 17, 0.65. Open green circles: 25 of the new points the error on new points is measured on

  • The fitted polynomial, no penalty: coefficients up to 684,000, ‖w‖ = 1.1 million

Regularization 1: a penalty on the weights

L(w)  =  Ltrain(w)⏟fit  +    λ∥w∥2⏟strength    ∥w∥2⏟size of the weights\mathcal{L}(\textcolor{#B5396B}{\mathbf{w}}) \;=\; \underbrace{\mathcal{L}_{\text{train}}(\textcolor{#B5396B}{\mathbf{w}})}_{\text{fit}} \;+\;\; \underbrace{\lambda\vphantom{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}}_{\text{strength}}\;\;\underbrace{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}_{\text{size of the weights}}

Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7

The same four panels with panel 2 highlighted and its fitted polynomial written out, y equals w 0 plus w 1 x up to w 9 x to the ninth, with the coefficients; the other panels dimmed. Four panels of the degree-9 polynomial fitted to the same 10 points, with an increasing penalty. No penalty: through every point, swinging far off, error on new points 0.86. Lambda 10 to the minus 7: a smooth curve close to the dashed one, 0.34. Lambda 10 to the minus 3: smooth, 0.34. Lambda 1: an almost flat line, training error 0.49, error on new points 0.61. Under each panel the size of the coefficient vector: 1.1 million, 189, 17, 0.65. Open green circles: 25 of the new points the error on new points is measured on

  • The fitted polynomial, λ = 10⁻⁷: coefficients under 100, ‖w‖ = 189

Regularization 1: a penalty on the weights

L(w)  =  Ltrain(w)⏟fit  +    λ∥w∥2⏟strength    ∥w∥2⏟size of the weights\mathcal{L}(\textcolor{#B5396B}{\mathbf{w}}) \;=\; \underbrace{\mathcal{L}_{\text{train}}(\textcolor{#B5396B}{\mathbf{w}})}_{\text{fit}} \;+\;\; \underbrace{\lambda\vphantom{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}}_{\text{strength}}\;\;\underbrace{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}_{\text{size of the weights}}

Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7

The same four panels with panel 3 highlighted and its fitted polynomial written out, y equals w 0 plus w 1 x up to w 9 x to the ninth, with the coefficients; the other panels dimmed. Four panels of the degree-9 polynomial fitted to the same 10 points, with an increasing penalty. No penalty: through every point, swinging far off, error on new points 0.86. Lambda 10 to the minus 7: a smooth curve close to the dashed one, 0.34. Lambda 10 to the minus 3: smooth, 0.34. Lambda 1: an almost flat line, training error 0.49, error on new points 0.61. Under each panel the size of the coefficient vector: 1.1 million, 189, 17, 0.65. Open green circles: 25 of the new points the error on new points is measured on

  • The fitted polynomial, λ = 10⁻³: coefficients under 12, ‖w‖ = 17

Regularization 1: a penalty on the weights

L(w)  =  Ltrain(w)⏟fit  +    λ∥w∥2⏟strength    ∥w∥2⏟size of the weights\mathcal{L}(\textcolor{#B5396B}{\mathbf{w}}) \;=\; \underbrace{\mathcal{L}_{\text{train}}(\textcolor{#B5396B}{\mathbf{w}})}_{\text{fit}} \;+\;\; \underbrace{\lambda\vphantom{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}}_{\text{strength}}\;\;\underbrace{\lVert\textcolor{#B5396B}{\mathbf{w}}\rVert^2}_{\text{size of the weights}}

Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7

The same four panels with panel 4 highlighted and its fitted polynomial written out, y equals w 0 plus w 1 x up to w 9 x to the ninth, with the coefficients; the other panels dimmed. Four panels of the degree-9 polynomial fitted to the same 10 points, with an increasing penalty. No penalty: through every point, swinging far off, error on new points 0.86. Lambda 10 to the minus 7: a smooth curve close to the dashed one, 0.34. Lambda 10 to the minus 3: smooth, 0.34. Lambda 1: an almost flat line, training error 0.49, error on new points 0.61. Under each panel the size of the coefficient vector: 1.1 million, 189, 17, 0.65. Open green circles: 25 of the new points the error on new points is measured on

  • The fitted polynomial, λ = 1: every coefficient under 0.35, ‖w‖ = 0.65, and the curve is nearly flat

Ridge regression: the penalty on a neuron model

Two panels for one monkey IT recording site and 512 features of the trained network's late layer. Left: predicted against recorded responses for a ridge fit; training images faint magenta, held-out images green, both scattered along the diagonal, R-squared 0.71 on training and 0.69 on held-out images. Right: R-squared against the penalty lambda on a log axis; with almost no penalty, training R-squared is 0.85 and held-out R-squared 0.28; as the penalty grows, training R-squared falls while held-out R-squared rises to meet it near 0.69 at the penalty chosen on the training images, then both fall

Computed: one monkey IT site (Bao et al. 2020), 1,224 object images, 512 late-layer units of ResNet-18; a random fifth held out

  • Encoding model: one IT site from 512 units by ridge regression, λ by cross-validation inside the training images (RidgeCV); too small a λ overfits (held-out R2R^2 0.28 vs 0.69)
  • You fit this in the second assignment: one IT site, one layer. Assignment 3 scores such models of IT sites for two ResNet-50s, and the model–brain lecture runs it for every block

Regularization 2: dropout

Three copies of the same small network: in the first two, a different random half of the hidden units is switched off for one training step each, with their connections removed; in the third, at test time, all units are on

  • Dropout: at each training step, switch off a random half of the hidden units; all are on at test time
  • Often with the weight penalty, sometimes L1 (absolute values): each cuts the freedom to fit noise

Regularization 3: more examples

Computed: the same curve and noise, after Bishop (2006), Fig. 1.6

Four panels of the same degree-9 polynomial fitted to more and more noisy points from the dashed sine-shaped curve. N equals 10: the curve swings far off, error on new points 0.86. N equals 15: close to the dashed curve, 0.34. N equals 30: 0.35. N equals 100: the fitted curve follows the dashed one, training error 0.28, error on new points 0.31. Under each panel the size of the coefficient vector: 1.1 million, 46,000, 37,000, 8,700. Open green circles: 25 of the new points the error on new points is measured on

  • The same degree-9 polynomial on 10, 15, 30, 100 points: test RMSE 0.86 → 0.31, ‖w‖ from a million to 8,700. More examples always help the same model

Regularization 3: more examples

Computed: the same curve and noise, after Bishop (2006), Fig. 1.6

The same four panels with panel 1 highlighted and its fitted polynomial written out with the coefficients; the other panels dimmed. Four panels of the same degree-9 polynomial fitted to more and more noisy points from the dashed sine-shaped curve. N equals 10: the curve swings far off, error on new points 0.86. N equals 15: close to the dashed curve, 0.34. N equals 30: 0.35. N equals 100: the fitted curve follows the dashed one, training error 0.28, error on new points 0.31. Under each panel the size of the coefficient vector: 1.1 million, 46,000, 37,000, 8,700. Open green circles: 25 of the new points the error on new points is measured on

  • The fitted polynomial, N = 10: the same coefficients as the unpenalized fit, ‖w‖ = 1.1 million

Regularization 3: more examples

Computed: the same curve and noise, after Bishop (2006), Fig. 1.6

The same four panels with panel 2 highlighted and its fitted polynomial written out with the coefficients; the other panels dimmed. Four panels of the same degree-9 polynomial fitted to more and more noisy points from the dashed sine-shaped curve. N equals 10: the curve swings far off, error on new points 0.86. N equals 15: close to the dashed curve, 0.34. N equals 30: 0.35. N equals 100: the fitted curve follows the dashed one, training error 0.28, error on new points 0.31. Under each panel the size of the coefficient vector: 1.1 million, 46,000, 37,000, 8,700. Open green circles: 25 of the new points the error on new points is measured on

  • The fitted polynomial, N = 15: ‖w‖ = 46,000

Regularization 3: more examples

Computed: the same curve and noise, after Bishop (2006), Fig. 1.6

The same four panels with panel 3 highlighted and its fitted polynomial written out with the coefficients; the other panels dimmed. Four panels of the same degree-9 polynomial fitted to more and more noisy points from the dashed sine-shaped curve. N equals 10: the curve swings far off, error on new points 0.86. N equals 15: close to the dashed curve, 0.34. N equals 30: 0.35. N equals 100: the fitted curve follows the dashed one, training error 0.28, error on new points 0.31. Under each panel the size of the coefficient vector: 1.1 million, 46,000, 37,000, 8,700. Open green circles: 25 of the new points the error on new points is measured on

  • The fitted polynomial, N = 30: ‖w‖ = 37,000

Regularization 3: more examples

Computed: the same curve and noise, after Bishop (2006), Fig. 1.6

The same four panels with panel 4 highlighted and its fitted polynomial written out with the coefficients; the other panels dimmed. Four panels of the same degree-9 polynomial fitted to more and more noisy points from the dashed sine-shaped curve. N equals 10: the curve swings far off, error on new points 0.86. N equals 15: close to the dashed curve, 0.34. N equals 30: 0.35. N equals 100: the fitted curve follows the dashed one, training error 0.28, error on new points 0.31. Under each panel the size of the coefficient vector: 1.1 million, 46,000, 37,000, 8,700. Open green circles: 25 of the new points the error on new points is measured on

  • The fitted polynomial, N = 100: ‖w‖ = 8,700; with more points the curve stops swinging between them

No more data? Make altered copies of it

Eight versions of one photograph of a pink orchid, each made by a torchvision transform: original, horizontal flip, resized crop, color jitter, rotation, affine warp, perspective, random erasing with a gray patch

  • Data augmentation: train on altered copies that keep the label: flips, crops, color, rotation, warps, erased patches. Choose transformations that should not change the label

Augmentation in PyTorch: a transform pipeline

transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.5, 1.0)),  # Crop and resize
    transforms.RandomHorizontalFlip(),  # Flip horizontally at random
    transforms.RandomRotation(15),  # Rotate by up to 15 degrees
    transforms.ColorJitter(0.3, 0.3, 0.3, 0.05),  # Vary color and brightness
    transforms.ToTensor(),  # Convert to channel-first floats in [0, 1]
    transforms.Normalize(IMAGENET_MEAN, IMAGENET_STD),  # Standardize channels
])
  • Each epoch draws a fresh random version of every image, so the network rarely sees the same picture twice. Flips that would change the label stay out: a 6 is not a 9

From the multilayer perceptron to the convolutional network

A dense layer on an image needs a weight for every pixel, and an object can appear anywhere: a small filter, applied at every position

Dense versus convolutional: where the weights go

Two panels over the same 6 by 6 image of pixels. Left, dense: every pixel is wired to every one of six hidden units, a tangle of 216 thin lines, and every hidden unit to every output unit; caption: every pixel to every unit, 36 times 6 weights in the first layer, all different. Right, convolutional: a 4 by 4 map of hidden units, three of which are shown wired each to its own 3 by 3 patch of pixels with the same nine color-coded weights, and one second-layer unit wired to a 2 by 2 patch of the map; caption: each unit sees a 3 by 3 patch, the nine weights are shared by all sixteen units

  • The perceptron wires every pixel to every unit. The convolutional layer wires each unit to a patch, and every unit shares the same weights: a feature detector copied across the image

Count the weights: a dense layer on an image

Horizontal bars on a log axis: 96 filters of 11 by 11 by 3, reused at every position, 34,848 weights; 96 dense units, each wired to every pixel of a 224 by 224 color image, 14,450,688 weights
  • Flatten a 224 × 224 color image: one dense unit needs 224 · 224 · 3 = 150,528 weights, and it ignores which pixels are neighbors
  • AlexNet's first layer instead learns 96 small filters of 11 × 11 × 3 and reuses each one at every position: 34,848 weights, about 400 times fewer than 96 dense units
  • An object can sit anywhere in the image, so the detector is needed at every position: local filters, the same everywhere

Try it: template matching at every position

  • The image is a grid of numbers; cut a 5 × 5 template, slide it: a dot product at each position. The best match is where it came from

Try it: slide the filter yourself

  • One 7 × 7 filter, the tiger in 96 × 96 gray levels: step the window, read the dot product, watch the map fill in; then switch the filter

Convolution: one filter, slid over the image

  • A filter (or kernel), a small grid of weights, a template, slid over the image; at each position the dot product, then ReLU: a feature map. Shift the image and the map shifts with it

The tiger photograph in gray levels, 224 by 224, an arrow to one 11 by 11 vertical Gabor filter, an arrow to the feature map, 224 by 224, bright where the image has vertical stripes

One filter gives one map; a layer gives a stack of maps

  • Four filters, four feature maps: the layer's output is a stack, one channel per filter, 224 × 224 × 4. Channels are the depth of the next layer's input

The same tiger at the same size, an arrow to four 11 by 11 Gabor filters at four orientations stacked as cards, an arrow to the four feature maps stacked as cards, all in the same color map as the single map before

Computed: the tiger, four 11 × 11 Gabor filters, stride 1, padding 5, ReLU

The convolutional layer, in notation

a[l]=g(W[l]a[l−1]+b[l])\mathbf{a}^{[l]} = g(\mathbf{W}^{[l]}\mathbf{a}^{[l-1]} + \mathbf{b}^{[l]})

A[l]=g(K[l]∗A[l−1]+b[l])\fmap{l} = \textcolor{#7A4FA3}{g}\big(\textcolor{#B5396B}{\kernel{l}} \conv \fmap{l-1} + \textcolor{#3F7A6B}{\mathbf{b}^{[l]}}\big)

Two panels over the same 6 by 6 image. Left, dense: every pixel wired to every hidden unit, a tangle of lines, the multilayer perceptron layer W a plus b. Right, convolutional: three hidden units of a 4 by 4 map each wired to its own 3 by 3 patch with the same nine color-coded weights, the convolution K star A plus b

  • Left, the MLP layer: every input to every unit. Right, the convolution: each unit sees a k×kk \times k patch of all Cl−1C_{l-1} input maps, the same weights at every position
  • K[l]\textcolor{#B5396B}{\kernel{l}}: the filters, ClC_l of them, each k×k×Cl−1k \times k \times C_{l-1}; A[l]\fmap{l}: the ClC_l feature maps, one channel per filter

Which brain areas turn an image into an object?

Simple and complex cells: the weighted sum and the pooling of the last slides, found in cortex decades earlier

Core object recognition: animal or not?

Monkeys do it too

  • Monkeys trained on the task answer as fast as people. From a few electrodes over V4 and posterior IT, a linear readout recovers the category, and it errs on the same images as the monkeys

The ventral stream: a hierarchy of areas

  • Tuning gets more complex (edges, parts, objects) and more tolerant of position and size
  • In IT, a linear readout of ~100 neurons reports the category: a linear classifier separates the categories
The ventral-stream hierarchy from retina and LGN through V1, V2, V4 to IT, with latencies growing from about 40 to 100 ms and neuron counts per stage

Retina and LGN: Kuffler's center–surround cells

Stephen Kuffler
Stephen Kuffler
  • A cell's receptive field: the patch of the visual field where light changes its firing. For a retinal ganglion or LGN cell, light on the + center excites, on the − surround inhibits (left: on-center, and its mirror image, off-center); uniform light does nothing. Right, Kuffler's records: these cells report local contrast, not brightness
Two schematic receptive fields: an on-center cell, a disc of plus signs inside a ring of minus signs, and an off-center cell, the reverse; no labels Kuffler 1953 as redrawn in Hubel's Eye, Brain and Vision: for an on-center retinal ganglion cell, four stimuli and their spike records: a small spot in the center gives a burst while the light is on; a larger spot covering center and surround gives little; a ring on the surround silences the cell and gives a burst when the light goes off; diffuse light over the whole field gives almost nothing

V1: Hubel & Wiesel's recording, on film

David Hubel and Torsten Wiesel at their recording setup
Hubel & Wiesel
  • One neuron in cat V1, one electrode, a bar of light moved by hand: the clicks are its spikes, and they come only for the right orientation at the right place

V1: Hubel & Wiesel's oriented cells

David Hubel and Torsten Wiesel at their recording setup
Hubel & Wiesel
  • Hubel & Wiesel mapped V1 cells by hand (left; crosses excite, triangles inhibit). Right, the same maps a–g as + and − regions: center–surround (a, b), bar detectors (c, d, e, f) and an edge detector (g), each at one orientation
Hubel and Wiesel 1962, receptive fields of cat cortical cells mapped by hand with spots and bars of light: crosses mark positions where light excites the cell, triangles where it inhibits; panels a and b are circular center-surround maps, c to g are elongated maps with an ON stripe flanked by OFF regions at one orientation The same seven receptive fields a to g redrawn with plus and minus regions: a, an on-center disc; b, an off-center disc; c, d, e and f, a plus stripe between minus flanks along the diagonal; g, an edge, plus on one side and minus on the other

V1 in one picture: ganglion → simple → complex

  • Left to right, ganglion or LGN cells, simple cells, complex cells: center–surround → oriented at one position → oriented anywhere nearby: a weighted sum, then a pooling, the two operations every stage of today's models repeats

Three columns of icons: an on-center and an off-center circle; an oriented bar on a dark or light background at one position; an oriented bar with arrows showing it can move across the field

Mechanism: a row of LGN cells makes a simple cell

  • Bar along the row: every center lit, the simple cell fires hard

Hubel and Wiesel's wiring model, computed on model cells, bar bar along the row: four on-center LGN receptive fields in a row on a screen, each with a plus center and a minus surround, a yellow bar of light across them, four LGN cell bodies with spike trains below, all converging on one cortical simple cell with its own spike train

Mechanism: a row of LGN cells makes a simple cell

  • Bar at 15°: the end cells lose their light; the simple cell nearly stops

Hubel and Wiesel's wiring model, computed on model cells, bar 15 degrees: four on-center LGN receptive fields in a row on a screen, each with a plus center and a minus surround, a yellow bar of light across them, four LGN cell bodies with spike trains below, all converging on one cortical simple cell with its own spike train

Mechanism: a row of LGN cells makes a simple cell

  • Bar at 30°: only the middle two centers lit; the simple cell is silent

Hubel and Wiesel's wiring model, computed on model cells, bar 30 degrees: four on-center LGN receptive fields in a row on a screen, each with a plus center and a minus surround, a yellow bar of light across them, four LGN cell bodies with spike trains below, all converging on one cortical simple cell with its own spike train

Mechanism: a row of LGN cells makes a simple cell

  • Bar at 60°: the bar crosses one center at a time

Hubel and Wiesel's wiring model, computed on model cells, bar 60 degrees: four on-center LGN receptive fields in a row on a screen, each with a plus center and a minus surround, a yellow bar of light across them, four LGN cell bodies with spike trains below, all converging on one cortical simple cell with its own spike train

Mechanism: a row of LGN cells makes a simple cell

  • Bar at 90°: no center lit, nothing fires

Hubel and Wiesel's wiring model, computed on model cells, bar 90 degrees: four on-center LGN receptive fields in a row on a screen, each with a plus center and a minus surround, a yellow bar of light across them, four LGN cell bodies with spike trains below, all converging on one cortical simple cell with its own spike train

Mechanism: a row of LGN cells makes a simple cell

  • Right orientation, shifted onto the surrounds: the LGN cells fall silent

Hubel and Wiesel's wiring model, computed on model cells, bar at the right orientation but shifted 0.5 units sideways onto the inhibitory surrounds: four on-center LGN receptive fields in a row on a screen, each with a plus center and a minus surround, a yellow bar of light across them, four LGN cell bodies with spike trains below, all converging on one cortical simple cell with its own spike train

Mechanism: a row of LGN cells makes a simple cell

  • Shifted further: the bar must line up with the + centers

Hubel and Wiesel's wiring model, computed on model cells, bar at the right orientation but shifted 0.8 units sideways onto the inhibitory surrounds: four on-center LGN receptive fields in a row on a screen, each with a plus center and a minus surround, a yellow bar of light across them, four LGN cell bodies with spike trains below, all converging on one cortical simple cell with its own spike train

Mechanism: simple cells converge on a complex cell

  • Bar left of all five fields: no cell responds

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all three converging symmetrically on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

Mechanism: simple cells converge on a complex cell

  • Bar between cells 1 and 2: both respond, and so does the complex cell

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all three converging symmetrically on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

Mechanism: simple cells converge on a complex cell

  • Bar between cells 2 and 3: both respond; the complex cell keeps firing

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all three converging symmetrically on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

Mechanism: simple cells converge on a complex cell

  • Bar centered on cell 3's field: it responds most, and so does the complex cell

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all three converging symmetrically on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

Mechanism: simple cells converge on a complex cell

  • Bar between cells 3 and 4: both respond

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all three converging symmetrically on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

Mechanism: simple cells converge on a complex cell

  • Bar between cells 4 and 5: the complex cell still fires

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all three converging symmetrically on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

Mechanism: simple cells converge on a complex cell

  • Past the fields: each simple cell covers a narrow range, the complex cell a broad one

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all three converging symmetrically on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

Recap

  • Generalize better: a penalty on the weights (weight decay, ridge), dropout, more or altered data, with λ chosen on validation data
  • Brain: center–surround, simple cells wired from rows of LGN cells, complex cells that pool them; receptive fields mapped by hand or by reverse correlation
  • Network: convolution (one filter, shared across positions) is the simple cell's weighted sum; pooling over nearby positions is the complex cell

Further reading for the research trail

Optional: a few research-trail entries over the semester. Any paper mentioned in any lecture qualifies.

Further reading, continued

Minute paper

Submit on Canvas → Minute papers → Minute Paper 12 (access code read out in class).

Write three brief points in your own words:

  1. Hubel and Wiesel's simple and complex cells each correspond to one operation of a convolutional
    network. Name the two operations, and say what each buys the network.
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

- Exam 1 is Thursday in class; tonight's recitation is exam preparation. The third assignment is released on Thursday

- Last lecture stopped after shortcut learning. These are the slides it did not reach: it showed when a fit fails to carry over to new data, and these say what to do about it - Regularization is the umbrella term. The next slides show three kinds: a penalty on the weights (also called weight decay or L2 regularization), dropout, and more examples, real or made by altering the ones you have. Early stopping, from the learning curves, is a fourth - Then the weight penalty applied to a model of a neuron, ridge regression, which the second assignment used and the next assignment uses again

- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect

- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect

- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect

- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect

- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect

- The weight penalty of the previous slides, applied to a model of a neuron instead of a curve - What is the same and what is new between the two assignments. Second assignment: one IT site from Bao et al. (2020), features from one layer of a network, nested cross-validation, scored by R². Next assignment: electrodes in V1, V4 and IT from the THINGS Ventral Stream Spiking Dataset (two monkeys, photographs), features from each of ResNet-50's 16 blocks, the same StandardScaler + RidgeCV with one λ per electrode, scored by the noise-corrected correlation (Pearson r divided by the square root of the electrode's reliability), and the question is which block predicts which area best - An encoding model predicts a neuron's response to each image from a network's features. With the penalty λ‖w‖² added to the squared error, the fit is called ridge regression. It is scored by R² on held-out images - The target is one IT site's response to each of the 1,224 object images of Bao et al. (2020); the features are the 512 units of the trained ResNet-18's late layer. A random fifth of the images is held out and never seen by the fit - The model is a weighted sum of the features plus a bias, fit by least squares: linear regression. Run with a small enough step on a continuous target, the delta rule minimizes the same squared error - λ is chosen by leave-one-out cross-validation within the 980 training images (scikit-learn's RidgeCV). The held-out fifth only scores the chosen fit. The held-out curve on the right is drawn to show what the penalty does; it was not used to choose λ - R² = 1 − (squared error of the prediction) / (squared error of predicting the mean response). 1 is perfect, 0 is no better than the mean, and it can be negative. The baselines lecture used the squared correlation, which is close for a good fit - Right panel: with almost no penalty, 512 weights fit the training images closely (R² 0.85) and predict held-out images poorly (0.28): overfitting. As λ grows, the training fit gets worse and the held-out fit better, up to a point (0.69) - The second assignment fit this model with nested cross-validation: λ is chosen inside each training fold, and the held-out R² comes from the outer folds - The same model, the same `RidgeCV`, twice. In the second assignment the target was one IT site and the features one network layer; in the next, the targets are recording sites in three areas and the features come from each of 16 stages of a network, so the fit is repeated 16 times per site - One λ per site: each site gets its own penalty, chosen on its own training images. RidgeCV does this with alpha_per_target=True - Nested cross-validation keeps the two decisions apart: λ is chosen on training folds; the score comes from images that were not used to choose anything

- Dropout (Srivastava et al. 2014): each hidden unit is switched off with probability p (often one half) at every training step, a different random set each time. A unit cannot count on any particular other unit, so the network cannot build fragile combinations that fit single training images. At test time all units are on, with their outputs scaled so their average matches training. PyTorch does the scaling during training instead (kept units × 1/(1−p)), so in evaluation mode nothing is rescaled. AlexNet used dropout in its fully connected layers; without it, Krizhevsky et al. (2012) wrote, the network "exhibits substantial overfitting" - L1 penalizes the sum of the absolute values of the weights instead of their squares. It drives many weights to exactly zero, so the model uses fewer inputs - The penalties differ in detail, but they share one effect: they restrict which weight settings training can reach, so the network has fewer ways to fit the noise in its training data - In PyTorch: nn.Dropout(p=0.5) as a layer; model.train() switches it on and model.eval() switches it off. Weight decay is the weight_decay argument of the optimizer

- Last lecture: the same **degree-9 polynomial** fitted to 10, 15, 30 and 100 points; test RMSE 0.86 → 0.31 - With 10 points it passes through every one; with 100 it cannot, and follows the curve instead - <mark>More examples always help the same model</mark>: overfitting is flexibility relative to the number of examples; $\lVert\mathbf{w}\rVert$ falls from a million to 8,700 with no penalty at all - The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had

- The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had

- The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had

- The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had

- The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had

- Labeled data is expensive, so the next best thing to more data is to make more from what you have - Each epoch shows a different random version of each image, which makes memorizing the training images harder - Flips that would change the label (a 6 and a 9, a b and a d) must be left out. Speech recognizers train on recordings with stretches of time and bands of frequency masked out (SpecAugment, Park et al. 2019); medical-image networks on rotated and re-lit scans - Krizhevsky et al. (2012) trained AlexNet on random 224 × 224 crops of each 256 × 256 photo and their mirror images, which multiplies the training set by 2,048, plus color changes - Photograph: THINGSplus, public domain (CC0)

- torchvision.transforms.Compose applies the list in order to each image as it is loaded; the random transforms draw new parameters every time, so the training set is effectively unbounded. The held-out and test loaders use only Resize, CenterCrop, ToTensor and Normalize - Krizhevsky et al. (2012) used random 224 × 224 crops of each 256 × 256 photo and their mirror images, which multiplies the training set by 2,048, plus color changes. The next assignment's fine-tuning uses a pipeline like this one - Augmentations that change the label must be left out: a flip turns a b into a d and a 6 into a 9. Speech recognizers mask stretches of time and frequency (SpecAugment); medical-image networks rotate and re-light scans

- The networks of the last three lectures connected every unit to every input. For images that is wasteful, and it misses something: an object can appear anywhere. The next slides turn the dense layer into a convolution, the operation every vision network since has used

- Left: a two-layer perceptron over a 6 × 6 image: every pixel to every hidden unit (216 weights, all different), every hidden unit to every output. Right: a convolutional layer: each hidden unit reads a 3 × 3 patch, and the nine weights (color-coded) are the same for every unit, so the layer has nine weights instead of 216 and detects the same feature wherever it appears. A second convolutional layer reads patches of the map in the same way - Local connections and shared weights are the two changes that turn the perceptron into a convolutional network; the next slide counts what that saves, and the ones after show it as one operation, the convolution

- Counts are weights only, without biases. The image size is the 224 × 224 × 3 input stated in Krizhevsky et al. (2012) - A dense (fully connected) layer is the MLP layer of the lecture on learning the weights: every unit has one weight for every input value - Reusing a filter is called weight sharing (also parameter sharing): the 96 filters are the only first-layer weights, however many positions they are applied at - A dense unit could learn a local pattern, but it would have to learn it separately at every position, from many more examples

- Retinotopic organization: neighboring points of the visual field are handled by neighboring cells, so cortex holds a map of the image, with the same local machinery everywhere - On a computer the same answer comes from one filter applied at every position in turn, so the weights exist once: weight sharing. Counts are weights only, without biases, for the 224 × 224 × 3 input of Krizhevsky et al. (2012): 96 × 11 × 11 × 3 = 34,848, against 96 × 150,528 = 14,450,688 for dense units - Both the template and each window have their mean removed and are scaled to unit length, so the dot product is a cosine: 1.00 at the position the template was cut from, lower elsewhere. The template-matching idea is the model neuron's: among inputs of the same length, wᵀx is largest for the input that looks like w

- The window's patch has its mean subtracted before the dot product, so the filter reports contrast, not brightness; ReLU sets negative values to 0 in the map - The vertical filter responds along the tiger's flank; the horizontal one along the water's edge; the center–surround filter marks spots and edges of any orientation - The map has 90 × 90 positions, the image 96 × 96: no padding, stride 1, as the next slides define - The demo is on the course site with the other demos

- A **filter** (or **kernel**): a small grid of weights, a template, like a **simple cell**'s receptive field. At each position, multiply by the pixels under it and sum; a **ReLU** sets negatives to 0 - The result is a **feature map**: how well each part of the image matches the filter - Same weights everywhere: shift the image, and the map shifts with it (**equivariance**) - Computed here: the tiger (gray levels, 224 × 224) correlated with one 11 × 11 Gabor filter at stride 1 and zero padding 5 (both defined on the stride-and-padding slide), then a ReLU. The circle marks the largest value of the map, at the boxed image position. This oriented filter responds along the tiger's stripes that share its angle - Each value of the map is one unit's response; all the units of one map share the filter's weights - Equivariance: the map moves when the image moves. Invariance: the output does not change at all. Pooling adds invariance, a few slides on - PyTorch's Conv2d computes this sliding dot product without flipping the filter, which mathematicians call cross-correlation; deep-learning papers call it convolution all the same

- One filter slid over the image gives one feature map. A layer has many filters, and their maps stack into a volume: height × width × channels, one channel per filter. The next layer's filters then read all channels at once, which is why their weights have a depth of C_{l−1} in the notation that follows - Computed here: four Gabor filters at 0°, 45°, 90°, 135° on the tiger in gray levels; each map lights up where the stripes of the image run along that filter's orientation

- $\fmap{l}$: the layer's **feature maps**, $C_l$ of them, one **channel** per filter; $H_l \times W^{\mathrm{sp}}_l$ is each map's height and width - $\textcolor{#B5396B}{\kernel{l}}$: <span style="color:#B5396B"><b>the layer's filters</b></span>, each $k \times k \times C_{l-1}$, one slice per input channel · $\conv$ slides them over $\fmap{l-1}$ · $\textcolor{#3F7A6B}{\mathbf{b}^{[l]}}$: <span style="color:#3F7A6B"><b>one bias per channel</b></span> · $\textcolor{#7A4FA3}{g}$: <span style="color:#7A4FA3"><b>ReLU</b></span> - It is the MLP layer $\mathbf{a}^{[l]} = g(\mathbf{W}^{[l]}\mathbf{a}^{[l-1]} + \mathbf{b}^{[l]})$ with each unit wired only to a $k \times k$ patch of all $C_{l-1}$ input maps, and the same weights at every position. Weights per layer: $k \cdot k \cdot C_{l-1} \cdot C_l$ - The input image is the first stack, A^[0], with three channels (red, green, blue) - One filter spans every input channel: a first-layer filter on a color image is k × k × 3; a filter in a layer whose input has 64 channels is k × k × 64. Each filter makes one output channel - Per element: A^[l] at row i, column j, channel c is g of (the sum over the filter's rows u, columns v and input channels c′ of K at u, v, c′, c times A^[l−1] at i+u, j+v, c′, plus b_c) - W^sp is the spatial width; the superscript keeps it apart from the weight matrix W - In PyTorch, nn.Conv2d(C_in, C_out, kernel_size=k) stores the filters as a tensor of shape (C_out, C_in, k, k)

- From here on, the brain: the behavior to explain, the areas from the retina to IT, and Hubel and Wiesel's cells, which turn out to compute the two operations of the convolutional network, a weighted sum and a pooling - V1 sees edges in small patches; people recognize whole objects in a glance. The rest of today asks what happens between the two, in the brain and in a network - The brain half follows the ventral stream: center–surround cells, then V1's simple and complex cells, then the areas above V1. On the way it asks a question about learning: V1's oriented receptive fields can be learned from natural images alone, with no labels, by a sparse code - The network half prepares the next assignment: convolution, pooling, residual networks, BatchNorm, and reusing a pretrained network

- Core object recognition: naming the object in a single glance, across changes in position, size and background (DiCarlo, Zoccolan & Rust 2012) - The movie: ten photographs from the animal / non-animal set of Serre, Oliva & Poggio (2007), five animals and five distractors at the four viewing distances; each is shown for two frames at 30 frames per second, about 67 ms, after a fixation cross, then a two-second window to answer, then the answer - Thorpe, Fize & Marlot (1996): photographs flashed for 20 ms, a go/no-go response to animals; 94% correct. The EEG over frontal cortex differed between animal and non-animal trials from about 150 ms after the image - Too fast for eye movements or many back-and-forth loops: recognition rides mostly on one feedforward sweep through the visual system. Feedback connections exist everywhere; the speed limits how much they can contribute to the first answer

- Cauchoix, Crouzet, Fize & Serre (2016): two monkeys trained on the rapid animal / non-animal task with the same photographs people see; intracranial electrodes over intermediate and higher ventral areas (V4, posterior IT). A linear readout of the recorded activity reported the category early enough to drive the behavioral response, and the images the readout got wrong were the ones the monkeys got wrong - The movie: one monkey at the screen, photographs flashed one after another, responses at the box

- The figure is DiCarlo, Zoccolan & Rust (2012), Fig. 1: the macaque ventral stream (A) and its areas stacked with their typical latencies and approximate neuron counts (B). AIT, CIT, PIT: anterior, central and posterior IT - LGN: lateral geniculate nucleus, the relay between retina and V1. IT: inferior temporal cortex - Hung, Kreiman, Poggio & DiCarlo (2005): about 100 IT neurons, over windows as short as 12.5 ms, carried enough information for a linear classifier to report object identity and category, and the readout generalized over positions and sizes - The geometry lecture called this untangling (DiCarlo & Cox 2007): in pixels two objects' images cannot be separated by a flat (linear) boundary; in IT they can - The question for the rest of the lecture: what operation, repeated, gets from edges to objects?

- Kuffler (1953) recorded retinal ganglion cells of the cat with small spots of light: an on-center cell fires to light in a small central region and is suppressed by light in the ring around it; off-center cells do the opposite. LGN cells, one synapse on, have the same organization - Because center and surround cancel, a uniform field does nothing and an edge or a spot does: the retina already computes contrast. These are the inputs Hubel and Wiesel's cortical cells are built from, on the next slides

- The original Hubel and Wiesel film from the late 1950s and 1960s, shown in the first lecture: the experimenter moves a projected bar of light over a screen while one neuron in cat primary visual cortex is recorded, its spikes played as clicks - The cell is silent for most positions and orientations and fires briskly for one orientation at one place: its receptive field and its orientation preference, found by hand before computers ran the experiments - Torsten Wiesel describes the experiments at iBiology; Hubel and Wiesel shared the 1981 Nobel Prize in Physiology or Medicine for this work

- Hubel and Wiesel (1962) mapped receptive fields in cat visual cortex with spots and bars projected on a screen, marking by hand where light excited the cell (crosses) and where it inhibited it (triangles). Circular maps (a, b) are the center–surround cells of the LGN and layer 4; the elongated ones (c–g) are simple cells, selective for the orientation of a bar - The schematic on the right is the textbook summary of such maps: an ON stripe between OFF flanks (or the reverse), one orientation per cell. A Gabor function, a grating inside a Gaussian window, fits these maps with a few numbers (orientation, stripe width, phase, window size; Jones and Palmer 1987); the second assignment fit Gabors to network units the same way - The cell's response falls off smoothly as a bar is rotated away from its preferred orientation: its orientation tuning curve, typically 20–40° wide at half height

- What Hubel and Wiesel found, in one picture, before the mechanisms: center–surround inputs from the retina and LGN, simple cells built by summing rows of those inputs, complex cells built by pooling simple cells of one orientation at nearby positions - Two operations, then: a weighted sum (selectivity, the simple cell) and a pooling (tolerance, the complex cell). A convolutional layer is the weighted sum applied at every position; max-pooling models the complex cell. The Neocognitron and HMAX repeat the pair up the hierarchy

- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights

- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights

- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights

- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights

- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights

- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights

- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights

- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks

- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks

- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks

- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks

- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks

- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks

- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks

- Today: regularization, the move from the perceptron to the convolutional network, and the visual system up to V1: center–surround cells, Hubel and Wiesel's simple and complex cells, and how each is wired - Lecture 11 stopped after the simple-to-complex pooling frames. Next lecture starts with the rest: the ice-cube model, the areas above V1, the Neocognitron and HMAX, more on convolution (several filters, stride and padding, pooling), then the slides already held over (the mini-HMAX, ResNet-50, reuse, what trained networks learn, scale), then explainability

- Yamins, Hong, Cadieu, Solomon, Seibert & DiCarlo (2014), Performance-optimized hierarchical models predict neural responses in higher visual cortex - Zhuang, Yan, Nayebi, Schrimpf, Frank, DiCarlo & Yamins (2021), Unsupervised neural network models of the ventral visual stream - Konkle & Alvarez (2022), A self-supervised domain-general learning framework for human ventral stream representation - Kar, Kubilius, Schmidt, Issa & DiCarlo (2019), Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior - Jinsi, Henderson & Tarr (2023), Early experience with low-pass filtered images facilitates visual category learning in a neural network model - Lu, Thorat, Cichy & Kietzmann (2026), Adopting a human developmental visual diet yields robust and shape-based AI vision - Also cited today and worth reading: Kuffler (1953) and Hubel & Wiesel (1962) on receptive fields, Fukushima (1980) on the Neocognitron, Serre, Oliva & Poggio (2007) on rapid categorization - Before posting, skim the existing entries on your paper; your post must add something not already said

- Second half of the reading list; any paper mentioned in any lecture qualifies for the research trail

- Submitted on Canvas (Minute papers → Minute Paper 12); credit for a thoughtful attempt