Models of the ventral stream, and what networks compute

Reader-friendly version of the 42 lecture slides: the lecture content in normal flow, with figure descriptions and data tables where the figure carries data. Open the presentation.

Slide 1

Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 12 · Theme 2: Learning representations: from neurons to deep networks

Models of the ventral stream, and what networks compute

Tuesday, October 27 · Fall 2026

Return to contents

Slide 2

Where we left off: one filter, slid over the image

The tiger photograph in gray levels, 224 by 224, an arrow to one 11 by 11 vertical Gabor filter, an arrow to the feature map, 224 by 224, bright where the image has vertical stripes

The tiger photograph in gray levels, 224 by 224, an arrow to one 11 by 11 vertical Gabor filter, an arrow to the feature map, 224 by 224, bright where the image has vertical stripes. One set of weights is reused at every position, so shifting the image shifts the map.

Open full-size figure

Return to contents

Slide 3

Where we left off: simple and complex cells

Three columns of icons: an on-center and an off-center circle; an oriented bar on a dark or light background at one position; an oriented bar with arrows showing it can move across the field

Three columns of icons summarizing Hubel and Wiesel's V1: on the left an on-center and an off-center circle for retinal and LGN cells; in the middle an oriented bar at one position for a simple cell; on the right an oriented bar with arrows showing it can sit anywhere in the field, for a complex cell. The point: a weighted sum builds selectivity, a pooling builds tolerance.

Open full-size figure

Return to contents

Slide 6

Complex cells, again: one orientation, anywhere nearby

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all converging on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

Computed on model cells. Left: the bar has passed all five fields. Right: five overlapping bumps, one per simple cell, each covering a narrow range of positions, and the complex cell's broad curve covering all five with small dips between them: tolerance to position, with the orientation preference of its inputs.

Open full-size figure

Return to contents

Slide 7

Cortical columns: every orientation, at every position

The ice-cube model of primary visual cortex: a block of cortex with layers I to VI, divided into left-eye and right-eye ocular dominance slabs, crossed by orientation columns color-coded blue, green, yellow and red for horizontal, oblique, vertical and the other oblique, with cytochrome-oxidase blobs as dashed cylinders

Hubel and Wiesel's ice-cube model of primary visual cortex (V1), as drawn in Kandel, Schwartz and Jessell's Principles of Neural Science. A block of cortex is shown in perspective, with cortical layers I to VI marked down its left side. The block is split into two halves, a left-eye and a right-eye ocular dominance slab. Its top surface is tiled with orientation columns, color-coded by the legend: blue for horizontal, green for one oblique, yellow for vertical and red for the other oblique. Dashed cylinders inside each slab are the blobs. The point: one small block holds every orientation for both eyes, and the block repeats across the whole visual field.

Open full-size figure
  • A column: a stack of cells through the six layers of cortex that prefer one orientation. Hubel and Wiesel's ice-cube model: a 1 mm block of V1 holds a column for every orientation, for each eye; the block repeats across the visual field
  • Nothing slides in cortex. Each position has its own cells, with their own synapses (connections, each with a weight). We call this replication; the network uses sharing
  • Both end up with the same template at every position. The network stores one filter; cortex has a separate copy at each position. How those copies come to match is not known
Hubel & Wiesel Proc R Soc B 1977 · Kandel, Schwartz & Jessell Principles of Neural Science 2000

Return to contents

Slide 8

The hierarchy continues beyond V1

The ventral-stream areas stacked as boxes, retina and LGN at the bottom, V1, V2, V4, posterior, central and anterior IT above, each with its typical response latency, about 40 milliseconds at the LGN rising to about 100 milliseconds in anterior IT, and an approximate neuron count per stage

DiCarlo, Zoccolan and Rust 2012, Figure 1, right panel: the ventral-stream areas drawn as stacked boxes, retina and LGN at the bottom, then V1, V2, V4, posterior, central and anterior IT, with arrows up between them. Beside each stage its typical response latency, about 40 milliseconds at the LGN, 50 in V1, 60 in V2, 70 in V4, 80 to 100 in IT, and an approximate neuron count, from about one million to tens of millions.

Open full-size figure
  • Within V1, Hubel and Wiesel found a hierarchy: center–surround → simple → complex, selectivity, then tolerance
  • The ventral stream: V1 → V2 → V4 → IT (inferotemporal cortex), the chain that ends in object recognition. The working assumption: the same two steps repeat in each area
  • Three kinds of evidence: anatomy orders the areas (next slide); cells prefer more complex things at each stage (the slides after); responses start later at each stage, about 40 ms at the LGN (the relay between eye and V1) to 100 ms in IT

Return to contents

Slide 9

Anatomy orders the areas: Felleman and Van Essen

Felleman and Van Essen's 1991 diagram of the macaque visual system: 32 cortical areas as colored boxes stacked in about ten levels from the retina and LGN at the bottom through V1, V2, V4 and the IT areas to area 46, TF, TH and the hippocampal areas ER and HC at the top, joined by hundreds of lines for their connections

Felleman and Van Essen's 1991 diagram of the macaque visual system, in its color rendering: 32 cortical areas as colored boxes stacked in about ten levels, from the retina (RGC) and LGN at the bottom, through V1 and V2 (split into M, P-B and P-I compartments), V3, VP, V4, MT and the parietal areas, to the IT areas (PITd, PITv, CITd, CITv, AITd, AITv), area 46, FEF, TF, TH and the hippocampal areas ER and HC at the top. Black and red lines join the areas, about 300 connections in all; the dorsal areas sit on the left, the ventral areas leading to IT on the right.

Open full-size figure
  • Felleman & Van Essen (1991): about 300 known connections between 32 visual areas
  • Cortex has six layers, and a connection's direction is written in them: feedforward axons (away from the eye) end in layer 4 of the next area; feedback axons (back toward V1) avoid layer 4
  • Ordering the areas so that every feedforward connection points up gives about ten levels, V1 at the bottom, IT, area 46 and the hippocampal areas at the top. Two main paths: the ventral stream to IT (what), the dorsal stream through MT to parietal cortex (where, how)

Return to contents

Slide 10

Receptive fields grow along the ventral stream

Kobatake and Tanaka 1994: receptive fields recorded along the ventral stream of the macaque drawn on the visual field, with lines to their recording sites on an outline of the cortex; small squares near the fovea for early areas, growing to fields many times the ten-degree bar across for anterior IT

Kobatake and Tanaka (1994): receptive fields recorded along the macaque ventral stream. In the center, an outline of the cortex with dots marking recording sites, from posterior on the left to anterior on the right. A line runs from each dot to the receptive field of the cells recorded there, drawn as a square or rectangle on a small cross that marks the center of gaze. Posterior sites, on the left, connect to small squares of a few degrees close to the cross; anterior sites in inferotemporal cortex (IT), on the right, connect to large squares, the largest several times the 10-degree scale bar in the top-right corner. The point: receptive fields grow along the hierarchy.

Open full-size figure

Return to contents

Slide 11

V4: beyond bars, non-Cartesian gratings

Three rows of three circular gratings: Cartesian stripes at three orientations; polar gratings, concentric rings, radial spokes and a spiral; hyperbolic gratings, saddle-shaped patterns at three rotations

Nine circular grating patches in three rows, drawn from the stimulus formulas. Cartesian: parallel stripes at three orientations. Polar: concentric rings, radial spokes, and a spiral. Hyperbolic: saddle-shaped patterns at three rotations. Gallant and colleagues found V4 cells that prefer the polar or hyperbolic patterns to any stripes.

Open full-size figure

Drawn from the stimulus formulas; schematic

  • Gallant, Braun & Van Essen (1993) showed V4 cells three families of gratings: the usual Cartesian stripes, polar ones (rings, spokes, spirals) and hyperbolic ones (saddles)
  • A minority of V4 cells responded best to a polar or hyperbolic grating, more than preferred Cartesian stripes: no single oriented bar detector prefers these patterns
  • Later studies of V4 found tuning for curvature, color and texture (next slides)

Return to contents

Slide 12

What the cells prefer, area by area

Kobatake and Tanaka's 1994 table of critical features by area, four columns: V2, a triangle and a bullseye; V4, stars, a cross, textured and curved shapes; posterior IT, a leaf shape, a cross, concentric rings, a half disc; anterior IT, striped bars, a dark blob, a face, a circle on a stem, a flask

Kobatake and Tanaka's 1994 table of critical features by cortical area, four columns. V2: a triangle and a bullseye. V4: a star, a circle with a dot, a textured bar, a cross, a dark square with a corner, concentric rings. Posterior IT: a leaf shape, a cross, concentric rings, a star, a half disc, a dotted bar, a striped half circle. Anterior IT: a striped block, a dark blob, concentric rings, a cartoon face, a square with a dark bar, a circle on a stem, a flask shape, a blob, a striped bar with a ball. Preferred stimuli grow from simple patterns to object parts and faces.

Open full-size figure

Return to contents

Slide 13

V4: curvature at an angular position

The 51 simple closed shapes of Pasupathy and Connor 2001, black silhouettes on a light background, from a circle and ovals through crescents and teardrops to three- and four-lobed shapes

The 51 simple closed shapes used by Pasupathy and Connor in 2001, drawn as black silhouettes in six rows: a dot and a disc, ovals, crescents, teardrops, three-lobed and four-lobed shapes, and rounded forms with one sharp point. One shape in the second row is outlined in red as the example used on the next panel.

Open full-size figure
One of the shapes with its boundary points marked by angular position around the center, 0, 45, 90, 135, 225, 270 and 315 degrees; a sharp convexity sits at 45 degrees

One of the shapes, a black form with two sharp convex points, drawn with gray axes through its center and red circles marking points on its boundary at angular positions 0, 45, 90, 135, 225, 270 and 315 degrees. A V4 cell's response is described by the curvature of the boundary at one such angular position.

Open full-size figure

Return to contents

Slide 14

V2: naturalistic textures

Nine texture discs in three rows: a honeycomb, a pile of pea pods and a perforated metal sheet. Left column, the original photographs; middle, spectrally matched noise, shapeless gray blotches; right, naturalistic textures that keep the cells, pods and holes as distorted but recognizable structure

Nine texture discs in three rows: a honeycomb, a pile of pea pods and a perforated metal sheet. Left column, the original photographs; middle, spectrally matched noise, shapeless gray blotches; right, naturalistic textures that keep the cells, pods and holes as distorted but recognizable structure. Freeman et al. 2013 made the middle column by matching only the statistics V1-like filters measure, and the right column by also matching the correlations between those filters. V2 neurons respond more to the right column than to the middle one; V1 neurons respond about equally.

Open full-size figure

Return to contents

Slide 15

Photographs and whole-stream recordings

Twenty THINGS photographs in a five by four grid, one per object concept: an aardvark, an accordion, bananas, a bike, a butterfly, a camel, a chair, a teapot, an elephant, a guitar, a hammer, candles, an owl, a pineapple, an umbrella, a violin, a zebra, an anchor, a cake and a hard hat

Twenty THINGS photographs in a five by four grid, one per object concept: an aardvark, an accordion, bananas, a bike, a butterfly, a camel, a chair, a teapot, an elephant, a guitar, a hammer, candles, an owl, a pineapple, an umbrella, a violin, a zebra, an anchor, a cake and a hard hat. THINGS has 1,854 object concepts and 26,107 photographs; these are from its CC0 subset (THINGSplus). TVSD showed 22,248 THINGS photographs to two monkeys while Utah arrays recorded V1, V4 and IT.

Open full-size figure
  • THINGS (Hebart et al. 2019): 1,854 object concepts, 26,107 photographs
  • TVSD (Papale et al. 2025): 22,248 THINGS photographs, two monkeys; Utah arrays (10 × 10 electrode grids) in V1, V4 and IT at once
  • Triple-N (Li et al. 2026): Neuropixels probes, hundreds of recording sites on one shank, across IT
  • NSD (Allen et al. 2022): 7 T fMRI in eight people, 73,000 photographs

Return to contents

Slide 16

From recordings to models

Stacking the simple and complex cell operations, area after area

Return to contents

Slide 17

Fukushima's Neocognitron (1980)

Fukushima 1980, Figure 2: the interconnections between layers, U0 to UC3, each layer a stack of cell planes k = 1 to K, each cell reading from a small patch of the layer before, the patches growing with depth

Fukushima (1980), Figure 2: the Neocognitron's layers drawn left to right as stacks of planes, the input layer U0, then US1, UC1, US2, UC2, US3 and UC3. Each layer holds several cell planes, labeled k = 1 at the top to k = K at the bottom of each stack. Lines from a small circle in one plane to a cell in the next layer show that each cell reads a small patch of the layer before; the planes shrink and the patches cover more of the input with depth. In the last layer, UC3, each plane is a single cell. The point: alternating S (template) and C (pooling) layers build a cell that responds to a whole pattern wherever it sits.

Open full-size figure
  • S-cells: simple cells, one template per plane (a plane is one feature map), at every position. C-cells: pool nearby S-cells of one plane
  • Repeat S then C; the top responds to a whole digit wherever it sits. Trained without labels, layer by layer

Return to contents

Slide 18

HMAX: a hierarchy constrained by physiology

Serre et al. 2007, Figure 1: on the left the wiring diagram of the primate visual system, dorsal and ventral streams from V1 through V2, V4, PIT and AIT to prefrontal cortex; in the middle the model, simple units (solid circles) and complex units (dashed circles) alternating from oriented S1 filters on a photograph of cows up to an animal versus non-animal classification unit, with bypass routes; on the right a table of the layers S1 to S4 and C1 to C3 with their receptive-field sizes from 0.2 to 7 degrees and their numbers of units from a million to a hundred

Serre, Wolf, Bileschi, Riesenhuber and Poggio (2007), Figure 1, in three parts. Left: a wiring diagram of the macaque visual system, boxes for visual areas joined by arrows, split by a dashed line into the dorsal 'where' stream and the ventral 'what' stream, with V1, V2, V4, PIT and AIT highlighted in color and prefrontal areas at the top. Middle: the model, rows of solid circles (simple units) and dashed circles (complex units) alternating from oriented filters on a photograph of cows, up through orange, green and blue units, to a purple 'animal vs. non-animal' unit; light blue lines are bypass routes. Right: a table of the model layers S1, C1, S2, C2, S2b, S3, C2b, C3 and S4, with receptive-field sizes growing from 0.2 to 1.1 degrees at S1 to 7 degrees at the top, and unit counts from 10 to the 7 down to 1 classification unit. The point: each stage of the model is matched to a stage of the ventral stream.

Open full-size figure
  • S and C alternate, V1 to IT. S units: template matching, a convolution, like a simple cell. C units: a max, like a complex cell, for tolerance to position and scale (next slide)
  • Hyperparameters (receptive-field sizes, tuning widths, pooling ranges) set by hand to match physiology, not fit to the task: "graduate student descent"
  • No backprop: higher S units learn by simple unsupervised, Hebbian-like rules, picking up frequent combinations of the C features below (from orientations to corners and curves); only the readout sees labels

Return to contents

Slide 19

Pooling over position, and over scale

HMAX's S1 Gabor filters in a grid: rows at 0, 45, 90 and 135 degrees; columns at sizes 7, 9, 11, 13, 35 and 37 pixels, each filter drawn at its true relative size. Magenta boxes group the sizes into scale bands 1 (7 and 9), 2 (11 and 13) and 8 (35 and 37), with an ellipsis for bands 3 to 7. Caption: C1 takes the max over two adjacent sizes and nearby positions

HMAX's S1 Gabor filters in a grid: rows at 0, 45, 90 and 135 degrees; columns at sizes 7, 9, 11, 13, 35 and 37 pixels, each filter drawn at its true relative size. Magenta boxes group the sizes into scale bands 1 (7 and 9), 2 (11 and 13) and 8 (35 and 37), with an ellipsis for bands 3 to 7. Caption: C1 takes the max over two adjacent sizes and nearby positions. In the full model there are 16 sizes, 7 by 7 to 37 by 37 in steps of 2, so 8 bands.

Open full-size figure
  • HMAX's S1 bank: Gabor filters at 4 orientations and 16 sizes. A C1 unit takes the max over nearby positions and over two adjacent sizes: 8 scale bands
  • So an object seen once at one size drives nearly the same C1 units at another size: HMAX has tolerance to scale built in
  • A network has no built-in scale pooling: its size tolerance comes from the sizes in its training photographs and from data augmentation, the rescaled copies of last lecture, and it fades outside the range it saw

Return to contents

Slide 20

Pooling: from a map to fewer numbers

A flower photograph and one of its C1 orientation maps, with an 8 by 8 area boxed. That area as a grid of its activation values, from 0.03 to 1, divided into 2 by 2 windows with the largest value in each window in bold. An arrow labeled 2 by 2 max, stride 2 leads to a 4 by 4 grid that holds only the winning values

A flower photograph and one of its C1 orientation maps, with an 8 by 8 area boxed. That area as a grid of its activation values, from 0.03 to 1, divided into 2 by 2 windows with the largest value in each window in bold. An arrow labeled 2 by 2 max, stride 2 leads to a 4 by 4 grid that holds only the winning values: the map is half as wide and half as tall.

Open full-size figure

Return to contents

Slide 21

The same two steps, higher up

Wiring from C1 to C2. Bottom: three pairs of C1 units (orange), each pair a vertical and a horizontal unit at one position. Middle: three S2 units (green), each a weighted sum of the pair below it and tuned to a corner. Top: one C2 unit (orange), the max over the three S2 units, tuned to the corner at any of the three positions

Wiring from C1 to C2. Bottom: three pairs of C1 units (orange), a vertical and a horizontal unit at each of three positions. Middle: three S2 units (green), each a weighted sum of the pair below it, tuned to a corner. Top: one C2 unit (orange), the max over the three S2 units: the corner at any of the three positions. Labels on the right name each stage.

Open full-size figure
  • A simple unit above V1 (HMAX's S2) is again a weighted sum, now over complex cells: it is selective for a conjunction of oriented edges at set positions, a corner, a curve, a textured patch
  • A complex unit above it (C2) pools those simple units over positions and sizes, so it responds to that conjunction anywhere in its field
  • Repeat a weighted sum, then pooling, and both selectivity and tolerance grow from V2 to V4 to IT

Return to contents

Slide 22

S2 and C2 on a real photograph

A black-eyed susan photograph passes through the HMAX stages from left to right. Four C1 maps, one per orientation, light up the petals that run in each direction. One S2 feature's template is tuned to 135-degree edges in its upper left and 45-degree edges along its bottom. Its S2 response map shows where that combination appears, with 6 by 6 pooling windows; pooling keeps one value per window, and C2, the maximum over all positions, is 0.689. A second row repeats this with the flower shifted 30 pixels up and 16 right: the S2 map hardly matches the first (correlation 0.04), yet C2 is 0.694, a change of 0.8 percent

A black-eyed susan photograph passes through the HMAX stages from left to right. Four C1 maps, one per orientation, light up the petals that run in each direction. One S2 feature's template is tuned to 135-degree edges in its upper left and 45-degree edges along its bottom. Its S2 response map shows where that combination appears, drawn with a grid of 6 by 6 pooling windows and a gold ring at its strongest response; pooling keeps one value per window, and C2, the maximum over all positions, is 0.689. A second row repeats this with the flower shifted 30 pixels up and 16 right: the S2 map hardly matches the first (correlation 0.04), yet C2 is 0.694, a change of 0.8 percent.

Open full-size figure

Return to contents

Slide 23

HMAX worked: predictions from V1 to behavior

Table from Serre and Poggio 2010: quantitative data compatible with the model, by area. Psychophysics: rapid animal categorization and the face inversion effect; LOC: face processing in fMRI; PFC: the differential role of IT and PFC in categorization; IT: tuning and invariance properties, read-out for object category, the average effect in IT; V4: the MAX operation, tuning for two-bar stimuli, two-spot interaction, tuning for boundary conformation, tuning for Cartesian and non-Cartesian gratings; V1: simple and complex cell tuning properties, the MAX operation in a subset of complex cells. Rows in blue are predictions the model made before the data, rows in red data consistent with the model, rows in black data used to set its parameters

Serre and Poggio (2010): a table of data that the HMAX model accounts for, with columns for brain area, type of data, and reference numbers for the biological and model data. Rows by area. Psychophysics: rapid animal categorization; face inversion effect. LOC: face processing (fMRI). PFC: differential role of IT and PFC in categorization. IT: tuning and invariance properties; read-out for object category; average effect in IT. V4: MAX operation; tuning for two-bar stimuli; two-spot interaction; tuning for boundary conformation; tuning for Cartesian and non-Cartesian gratings. V1: simple and complex cell tuning properties; MAX operation in a subset of complex cells. Text color marks each row's status: black, data used to set the parameters; red, data the model agrees with; blue, predictions made before the data. The point: one model with one set of parameters was tested from V1 to behavior.

Open full-size figure
  • One model, one set of parameters, tested from V1 to behavior
  • Some results were predicted before the data: max-like pooling in V4 and in some V1 complex cells
  • V4 tuning for curved boundaries (Pasupathy & Connor) was matched by fitting model units to each cell; a linear readout of the same model units named the category as well as a linear readout of recorded IT sites (Hung et al. 2005)

Return to contents

Slide 24

The test on behavior: rapid animal categorization

Performance in d-prime for the model, red solid line, and for human observers, blue dashed line, across four viewing distances, head, close-body, medium-body and far-body: both peak near 2.5 at close-body and fall to about 1.4 at far-body, with overlapping error bars

Serre, Oliva and Poggio (2007): performance on rapid animal versus non-animal categorization, measured as d-prime (d′, how well hits are separated from false alarms), from 1.0 to 2.6, at four viewing distances: head, close-body, medium-body and far-body. The model is a solid red line and human observers a blue dashed line, both with error bars. Both rise from head, about 2.2 for the model and 2.4 for people, to a peak near 2.5 at close-body, then fall to about 1.4 at far-body; the error bars overlap at every distance. The point: the feedforward model matches human accuracy across the four conditions.

Open full-size figure
The stimulus set: animals photographed at four viewing distances, head, close-body, medium-body and far-body, each with a matched distractor scene

The stimulus set of Serre, Oliva and Poggio (2007): eight grayscale photographs in two rows under four column labels, head, close-body, medium-body and far-body. The top row shows animals at increasing distance: a leopard's head, a deer grazing, a small animal in a dark field, and distant animals in a landscape. The bottom row shows matched distractor scenes without animals at the same scales: a flower, a rocky stream, rock formations, and an open field. The point: the task is to say whether an animal is present, at four viewing distances.

Open full-size figure
  • Serre, Oliva & Poggio (2007): the animal / non-animal task of the vision lecture's opening movie. Photographs flashed for 20 ms, then a noise mask (a second image that overwrites the first) 50 ms after onset: too soon for feedback from higher areas to return
  • d′d': sensitivity, hits against false alarms, 0 = guessing. The model matched people at every viewing distance and was wrong on the same images more often than two equally good observers would be
  • Conclusion: with an early mask, one feedforward pass is enough

Return to contents

Slide 25

A mini-HMAX in PyTorch

class MiniHMAX(nn.Module): def __init__(self, num_classes): super().__init__() self.s1 = nn.Sequential(nn.Conv2d(3, 16, kernel_size=5, padding=2), nn.ReLU()) self.c1 = nn.MaxPool2d(4) # max over 4 × 4 blocks self.s2b = nn.Sequential(nn.Conv2d(16, 64, kernel_size=7, padding=3), nn.ReLU()) self.c2b = nn.AdaptiveMaxPool2d(1) # max over all positions self.readout = nn.Linear(64, num_classes) def forward(self, images): # (batch, 3, 224, 224) hidden = self.s1(images) # (batch, 16, 224, 224) hidden = self.c1(hidden) # (batch, 16, 56, 56) hidden = self.s2b(hidden) # (batch, 64, 56, 56) pooled = self.c2b(hidden).flatten(1) # (batch, 64) return self.readout(pooled) # class scores (logits)

Return to contents

Slide 26

Then scale took over

ImageNet top-5 accuracy against year for five networks: AlexNet 2012, 83.6 percent with 8 layers; ZFNet 2013, 88.3 percent, 8 layers; GoogLeNet 2014, 93.3 percent, 22 layers; VGG-16 2014, 92.7 percent, 16 layers; ResNet-152 2015, 96.4 percent, 152 layers, highlighted

ImageNet top-5 accuracy against year for five networks, drawn with the layer count beside each point: AlexNet 2012, 83.6 percent, 8 layers; ZFNet 2013, 88.3 percent, 8 layers; GoogLeNet 2014, 93.3 percent, 22 layers; VGG-16 2014, 92.7 percent, 16 layers; ResNet-152 2015, 96.4 percent, 152 layers, highlighted. Accuracy rises every year as depth grows.

Open full-size figure
  • CNNs existed, LeCun's LeNet (1998), with every filter learned end to end by backprop; through the 2000s they and HMAX were about equally good, and neither got much better with more data
  • What changed in 2012: GPUs, and 1.2 million labeled photographs (ImageNet). AlexNet, then 8 → 152 layers in three years, better every year (top-5: the right label among the five best guesses)
  • AlexNet kept the same stack (a weighted sum, then a max) but learned every weight from the labels

Return to contents

Slide 27

Same operations, different settings and learning

Return to contents

Slide 28

AlexNet (2012): the first layer it learned

AlexNet's 96 first-layer filters from Krizhevsky et al. 2012, Figure 3: the top three rows mostly gray oriented edges and gratings at several sizes, the bottom three rows mostly color blobs and color edges

Krizhevsky, Sutskever and Hinton (2012), Figure 3: AlexNet's 96 first-layer filters, each 11 by 11 by 3, shown as small color tiles in six rows of sixteen. The top three rows are mostly gray-scale oriented edges and stripe patterns (gratings) at many orientations and widths. The bottom three rows are mostly color blobs and color edges, with a few checkerboard patterns and fine vertical stripes. The point: training on labeled images alone produced oriented, Gabor-like filters similar to V1 simple cells.

Open full-size figure

Return to contents

Slide 29

ResNet-50: four block groups

ResNet-50 from left to right: a 224 by 224 tiger image; conv1, a 7 by 7 convolution at stride 2, 112 by 112; max-pool, 56 by 56; layer1, 3 blocks, 56 by 56; layer2, 4 blocks, 28 by 28; layer3, 6 blocks, 14 by 14; layer4, 3 blocks, 7 by 7; global average, 1 by 1; readout

ResNet-50 drawn left to right as a chain of squares joined by arrows, each labeled with its map size. A 224 by 224 photograph of a tiger (the image), then conv1, a 7 by 7 convolution at stride 2, 112 by 112; max-pool, 56 by 56; layer1, 3 blocks, 56 by 56; layer2, 4 blocks, 28 by 28; layer3, 6 blocks, 14 by 14; layer4, 3 blocks, 7 by 7; global average, 1 by 1; and a readout box. The four block groups are outlined in magenta and the squares shrink from left to right. The point: 16 residual blocks in four groups, each group after the first halving the maps' height and width.

Open full-size figure

Return to contents

Slide 30

Freeze the encoder, or fine-tune the last block group

The same ResNet-50 diagram, now colored: conv1, max-pool and layer1 to layer3 in gray, frozen, weights kept; layer4 and the readout in magenta, trained on the new task, fine-tuned

The same ResNet-50 chain as the earlier diagram, from a 224 by 224 tiger photograph through conv1, max-pool, layer1 to layer4, global average and readout, with the same map sizes. Now colored by what is trained: conv1, max-pool and layer1 to layer3 are gray, labeled in the legend 'frozen: weights kept'; layer4 and the readout are solid magenta, 'trained on the new task (fine-tuned)'. The point: fine-tuning can keep the early block groups fixed and retrain only the last block group and the readout.

Open full-size figure

Return to contents

Slide 31

Adversarial examples

Goodfellow et al. 2015, Figure 1: a photograph of a panda, classified panda with 57.7 percent confidence, plus 0.007 times a colorful noise pattern, the sign of the gradient of the loss with respect to the pixels, equals an image that looks identical to the panda but is classified gibbon with 99.3 percent confidence

Goodfellow et al. 2015, Figure 1: a photograph of a panda, classified panda with 57.7 percent confidence, plus 0.007 times a colorful noise pattern, the sign of the gradient of the loss with respect to the pixels, equals an image that looks identical to the panda but is classified gibbon with 99.3 percent confidence.

Open full-size figure

Return to contents

Slide 32

Three ResNet-50s, one architecture, three trainings

The same ResNet-50 diagram as two slides ago: a tiger image, conv1, max-pool, four block groups layer1 to layer4, global average pooling and a readout; the architecture every one of the three networks shares

ResNet-50 drawn left to right as a chain of squares joined by arrows, each labeled with its map size. A 224 by 224 photograph of a tiger (the image), then conv1, a 7 by 7 convolution at stride 2, 112 by 112; max-pool, 56 by 56; layer1, 3 blocks, 56 by 56; layer2, 4 blocks, 28 by 28; layer3, 6 blocks, 14 by 14; layer4, 3 blocks, 7 by 7; global average, 1 by 1; and a readout box. The four block groups are outlined in magenta and the squares shrink from left to right. The point: 16 residual blocks in four groups, each group after the first halving the maps' height and width.

Open full-size figure
  • Supervised: ImageNet's 1.2 million labeled photographs; the network of the ResNet-50 slides
  • DINO (self-supervised): no labels; two augmented views of one photograph must agree
  • Robust: trained on adversarial examples (previous slide)
  • Same architecture, three sets of weights. In the assignment: saliency of the standard and robust networks

Return to contents

Slide 33

Recap

Return to contents

Slide 34

Further reading for the research trail

Optional: a few research-trail entries over the semester. Any paper mentioned in any lecture qualifies.

Return to contents

Slide 35

Minute paper

Submit on Canvas → Minute papers → Minute Paper 13 (access code read out in class).

Write three brief points in your own words:

  1. Name two of the three kinds of evidence that the ventral stream is organized as a hierarchy, and say in one sentence what each shows.
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents

Slide 36

Going further: more of the network

Optional reading: stride and padding, VGG, residual blocks, batch normalization, training and evaluation modes

Return to contents

Slide 37

Stride and padding set the size of a map

Two panels. Left: a 5 by 5 input surrounded by one ring of zeros, a 3 by 3 filter at its first position and, dashed, one step to the right, labeled stride 1; output 5 by 5. Right: the same input with the filter moved two steps, stride 2; output 3 by 3

Two panels showing how stride and padding set the size of a feature map. Left: a 5 by 5 input surrounded by one ring of zeros (padding 1), making a 7 by 7 grid. A 3 by 3 filter window, outlined in magenta, sits at the top-left; a dashed copy one column to the right is labeled stride 1. An arrow leads to the output, a 5 by 5 grid. Right: the same padded input, with the dashed window moved two columns, labeled stride 2; the output is 3 by 3. The point: a larger stride makes a smaller map, and padding lets the filter sit on the border.

Open full-size figure

Return to contents

Slide 38

VGG (2014): small filters, stacked

Left, one 7 by 7 filter: 49 weights, one ReLU. Right, three 3 by 3 filters in a row: the receptive field grows from 3 to 5 to 7 pixels, 27 weights, three ReLUs

Two diagrams. Left: a single 7 by 7 grid of weights, labeled one 7 by 7 filter: it sees 7 by 7 pixels, 49 weights, one ReLU. Right: three nested grids of 3, 5 and 7 cells, labeled three 3 by 3 filters in a row: the receptive field grows from 3 to 5 to 7 pixels with 27 weights and three ReLUs.

Open full-size figure

Return to contents

Slide 39

Deeper networks: add the input back

A residual block: the block input A passes through a 1 by 1 convolution with BatchNorm and ReLU, a 3 by 3 convolution with BatchNorm and ReLU, and a 1 by 1 convolution with BatchNorm, together labeled F of A, the change the block learns; a green skip connection carries A unchanged around them to a plus node, followed by a ReLU, giving the block output g of A plus F of A

A residual block drawn top to bottom. The block input, bold A, enters three stacked boxes: a 1 by 1 convolution with BatchNorm and ReLU, a 3 by 3 convolution with BatchNorm and ReLU, and a 1 by 1 convolution with BatchNorm. A magenta bracket labels the three boxes 'F of A: the change the block learns'. A green line, the skip connection, carries A unchanged around the boxes to a plus node, where it is added to F of A. A ReLU follows, and the block output is g of, A plus F of A. The point: the block learns only a change to its input, so the input can pass through when no change is needed.

Open full-size figure
  • Stacking more plain layers made networks harder to train: a 56-layer network reached higher training error than a 20-layer one
  • A residual block computes a change F(A)\mathcal{F}(\boldsymbol{A}) and adds it to its input: output g(A+F(A))g(\boldsymbol{A} + \mathcal{F}(\boldsymbol{A})). The added path is a skip connection
  • If a block is not needed, its change can stay near zero and the input passes through: networks of 50 to 152 layers trained well
  • Each convolution is followed by BatchNorm, explained on its own slide

Return to contents

Slide 40

Normalization, from cortex to networks

Response of a model neuron against the contrast of its preferred grating, on a log axis, for three masks. Without a mask the response rises from 0 near 1 percent contrast to almost 1 at 100 percent; a weak and then a strong mask shift the curve to the right, so the same grating drives a weaker response. Above the plot: response equals drive divided by sigma plus the pooled drive of neighbors

Response of a model neuron against the contrast of its preferred grating, on a log axis, for three masks. Without a mask the response rises from 0 near 1 percent contrast to almost 1 at 100 percent; a weak and then a strong mask shift the curve to the right, so the same grating drives a weaker response. Above the plot: response equals drive divided by sigma plus the pooled drive of neighbors. Computed from the normalization model of Heeger 1992 and Carandini and Heeger 2012.

Open full-size figure

Computed from the normalization model; schematic

  • Divisive normalization: a neuron's drive is divided by the pooled activity of its neighbors. A mask that drives the pool lowers the response at every contrast; found from the retina to cortex
  • HMAX's S1 divides each filter response by the energy of the image patch. AlexNet normalized across neighboring channels (local response normalization)
  • BatchNorm (2015, standard since ResNet) divides each channel by its spread over a minibatch: a simplified normalization, chosen because it speeds training

Return to contents

Slide 41

Batch normalization

Three histograms of one channel of ResNet-50's first layer over 64 photographs and every position. Before: mean minus 0.81, standard deviation 3.19, spread from about minus 10 to 10. After subtracting the batch mean and dividing by the batch standard deviation: mean 0, standard deviation 1. After the learned scale gamma and shift beta: mean 0.58, standard deviation 0.36

Three histograms stacked vertically, all on the same horizontal axis of activation value from minus 10 to 10, for one channel of ResNet-50's first layer over 64 Imagenette photographs and every position. Top, in blue, before normalization: mean minus 0.81, standard deviation 3.19, a sharp peak near zero with long tails out to about minus 10 and 10. Middle, in gray, after subtracting the batch mean and dividing by the batch standard deviation: mean 0.00, standard deviation 1.00. Bottom, in magenta, after the learned scale gamma and shift beta: mean 0.58, standard deviation 0.36, a narrow peak just above zero. A dashed line marks each mean. The point: BatchNorm standardizes each channel, then rescales it with learned values.

Open full-size figure

Computed: ResNet-50, 64 Imagenette photographs

  • As training changes the weights of earlier layers, the scale of each channel's values drifts. Batch normalization (BatchNorm) rescales every channel, layer after layer
  • For each channel: subtract the mean μB\mu_{\mathcal{B}} and divide by the standard deviation (s.d.) of the current minibatch (all its images and positions), then apply a learned scale γ\gamma and shift β\beta
  • z^=(z−μB) /σB2+ϵ,z~=γ z^+β\hat{z} = (z - \mu_{\mathcal{B}})\,/\sqrt{\sigma^2_{\mathcal{B}} + \epsilon}, \quad \tilde{z} = \gamma\,\hat{z} + \beta

Return to contents

Slide 42

Training mode and evaluation mode

Two rows. Training mode, model.train(): a minibatch of images gives the batch mean and variance, used to normalize before gamma and beta and the next layer; the batch statistics also update a stored running mean and variance. Evaluation mode, model.eval(): images to classify are normalized with the stored running mean and variance, then gamma and beta, then the next layer

A flow diagram in two rows. Top row, training mode, model.train(): a box 'a minibatch of images' leads to 'batch mean mu-B and variance sigma-B squared', then to 'normalize, then gamma and beta', then to the next layer. A blue arrow labeled 'updates' runs down from the batch statistics to a yellow box, 'stored running mean and variance'. Bottom row, evaluation mode, model.eval(): 'images to classify' leads straight to 'normalize, then gamma and beta' and the next layer, with a gray arrow labeled 'used' coming from the stored running mean and variance. The point: training normalizes with each minibatch's own statistics; evaluation uses the stored ones.

Open full-size figure

Return to contents