Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 12 · Theme 2: Learning representations: from neurons to deep networks

Models of the ventral stream, and what networks compute

Tuesday, October 27 · Fall 2026

Reader-friendly version — lecture content with figure descriptions

Where we left off: one filter, slid over the image

The tiger photograph in gray levels, 224 by 224, an arrow to one 11 by 11 vertical Gabor filter, an arrow to the feature map, 224 by 224, bright where the image has vertical stripes

  • A filter, a small grid of weights, a template like a simple cell's receptive field, is slid over the image; at each position the dot product, then ReLU: a feature map
  • One set of weights, reused at every position (weight sharing), so the layer has few weights. Shift the image and the map shifts with it

Where we left off: simple and complex cells

Three columns of icons: an on-center and an off-center circle; an oriented bar on a dark or light background at one position; an oriented bar with arrows showing it can move across the field

  • Hubel and Wiesel's two cell types in V1: the simple cell, a weighted sum, selective for an oriented bar at one position; the complex cell, which pools simple cells and responds to the same bar anywhere nearby

Complex cells, again: one orientation, anywhere nearby

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all converging on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

  • A complex cell sums five simple cells of one orientation, their fields side by side
  • Whichever simple cell the bar falls on, the complex cell fires
  • Tolerant to position, selective for orientation: this is pooling

Complex cells, again: one orientation, anywhere nearby

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all converging on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

  • A complex cell sums five simple cells of one orientation, their fields side by side
  • Whichever simple cell the bar falls on, the complex cell fires
  • Tolerant to position, selective for orientation: this is pooling

Complex cells, again: one orientation, anywhere nearby

Computed on model cells. Left: five identical simple cells in a row, their oriented receptive fields overlapping, a yellow bar of light at one position, spike trains next to each cell, all converging on one complex cell below, which sums them. Right: response against the bar's position, five overlapping bumps for the simple cells and the complex cell's broad curve, their sum, drawn up to the current position

  • A complex cell sums five simple cells of one orientation, their fields side by side
  • Whichever simple cell the bar falls on, the complex cell fires
  • Tolerant to position, selective for orientation: this is pooling

Cortical columns: every orientation, at every position

The ice-cube model of primary visual cortex: a block of cortex with layers I to VI, divided into left-eye and right-eye ocular dominance slabs, crossed by orientation columns color-coded blue, green, yellow and red for horizontal, oblique, vertical and the other oblique, with cytochrome-oxidase blobs as dashed cylinders
  • A column: a stack of cells through the six layers of cortex that prefer one orientation. Hubel and Wiesel's ice-cube model: a 1 mm block of V1 holds a column for every orientation, for each eye; the block repeats across the visual field
  • Nothing slides in cortex. Each position has its own cells, with their own synapses (connections, each with a weight). We call this replication; the network uses sharing
  • Both end up with the same template at every position. The network stores one filter; cortex has a separate copy at each position. How those copies come to match is not known
Hubel & Wiesel Proc R Soc B 1977 · Kandel, Schwartz & Jessell Principles of Neural Science 2000

The hierarchy continues beyond V1

The ventral-stream areas stacked as boxes, retina and LGN at the bottom, V1, V2, V4, posterior, central and anterior IT above, each with its typical response latency, about 40 milliseconds at the LGN rising to about 100 milliseconds in anterior IT, and an approximate neuron count per stage
  • Within V1, Hubel and Wiesel found a hierarchy: center–surround → simple → complex, selectivity, then tolerance
  • The ventral stream: V1 → V2 → V4 → IT (inferotemporal cortex), the chain that ends in object recognition. The working assumption: the same two steps repeat in each area
  • Three kinds of evidence: anatomy orders the areas (next slide); cells prefer more complex things at each stage (the slides after); responses start later at each stage, about 40 ms at the LGN (the relay between eye and V1) to 100 ms in IT

Anatomy orders the areas: Felleman and Van Essen

Felleman and Van Essen's 1991 diagram of the macaque visual system: 32 cortical areas as colored boxes stacked in about ten levels from the retina and LGN at the bottom through V1, V2, V4 and the IT areas to area 46, TF, TH and the hippocampal areas ER and HC at the top, joined by hundreds of lines for their connections
  • Felleman & Van Essen (1991): about 300 known connections between 32 visual areas
  • Cortex has six layers, and a connection's direction is written in them: feedforward axons (away from the eye) end in layer 4 of the next area; feedback axons (back toward V1) avoid layer 4
  • Ordering the areas so that every feedforward connection points up gives about ten levels, V1 at the bottom, IT, area 46 and the hippocampal areas at the top. Two main paths: the ventral stream to IT (what), the dorsal stream through MT to parietal cortex (where, how)

Receptive fields grow along the ventral stream

Kobatake and Tanaka 1994: receptive fields recorded along the ventral stream of the macaque drawn on the visual field, with lines to their recording sites on an outline of the cortex; small squares near the fovea for early areas, growing to fields many times the ten-degree bar across for anterior IT
  • From V1 to anterior IT, receptive fields grow from under a degree of visual angle (a thumbnail at arm's length) to tens of degrees, while the preferred stimulus changes from bars to objects

V4: beyond bars, non-Cartesian gratings

Three rows of three circular gratings: Cartesian stripes at three orientations; polar gratings, concentric rings, radial spokes and a spiral; hyperbolic gratings, saddle-shaped patterns at three rotations

Drawn from the stimulus formulas; schematic

  • Gallant, Braun & Van Essen (1993) showed V4 cells three families of gratings: the usual Cartesian stripes, polar ones (rings, spokes, spirals) and hyperbolic ones (saddles)
  • A minority of V4 cells responded best to a polar or hyperbolic grating, more than preferred Cartesian stripes: no single oriented bar detector prefers these patterns
  • Later studies of V4 found tuning for curvature, color and texture (next slides)

What the cells prefer, area by area

Kobatake and Tanaka's 1994 table of critical features by area, four columns: V2, a triangle and a bullseye; V4, stars, a cross, textured and curved shapes; posterior IT, a leaf shape, a cross, concentric rings, a half disc; anterior IT, striped bars, a dark blob, a face, a circle on a stem, a flask
  • Kobatake & Tanaka (1994) reduced each cell's best object to the simplest picture that still drove it. Most V2 cells settle for a bar or a spot; the share of cells that need more grows up the stream: curves and textured shapes in V4, parts in posterior IT, faces and hands in anterior IT

V4: curvature at an angular position

The 51 simple closed shapes of Pasupathy and Connor 2001, black silhouettes on a light background, from a circle and ovals through crescents and teardrops to three- and four-lobed shapes One of the shapes with its boundary points marked by angular position around the center, 0, 45, 90, 135, 225, 270 and 315 degrees; a sharp convexity sits at 45 degrees
  • Pasupathy & Connor (2001) showed V4 cells 51 shapes at 8 rotations and found that a cell's response is set by the curvature of the boundary at one angular position around the shape's center, a sharp convexity at 45°, say: a part of the outline, not the whole shape

V2: naturalistic textures

Nine texture discs in three rows: a honeycomb, a pile of pea pods and a perforated metal sheet. Left column, the original photographs; middle, spectrally matched noise, shapeless gray blotches; right, naturalistic textures that keep the cells, pods and holes as distorted but recognizable structure
  • Freeman et al. (2013): two synthetic versions of each photograph. Spectrally matched noise keeps the statistics of V1-like filters; naturalistic texture also keeps their correlations
  • V1 responds alike to both; V2 responds more to naturalistic texture, and so do people

Photographs and whole-stream recordings

Twenty THINGS photographs in a five by four grid, one per object concept: an aardvark, an accordion, bananas, a bike, a butterfly, a camel, a chair, a teapot, an elephant, a guitar, a hammer, candles, an owl, a pineapple, an umbrella, a violin, a zebra, an anchor, a cake and a hard hat
  • THINGS (Hebart et al. 2019): 1,854 object concepts, 26,107 photographs
  • TVSD (Papale et al. 2025): 22,248 THINGS photographs, two monkeys; Utah arrays (10 × 10 electrode grids) in V1, V4 and IT at once
  • Triple-N (Li et al. 2026): Neuropixels probes, hundreds of recording sites on one shank, across IT
  • NSD (Allen et al. 2022): 7 T fMRI in eight people, 73,000 photographs

From recordings to models

Stacking the simple and complex cell operations, area after area

Fukushima's Neocognitron (1980)

Fukushima 1980, Figure 2: the interconnections between layers, U0 to UC3, each layer a stack of cell planes k = 1 to K, each cell reading from a small patch of the layer before, the patches growing with depth
  • S-cells: simple cells, one template per plane (a plane is one feature map), at every position. C-cells: pool nearby S-cells of one plane
  • Repeat S then C; the top responds to a whole digit wherever it sits. Trained without labels, layer by layer

HMAX: a hierarchy constrained by physiology

Serre et al. 2007, Figure 1: on the left the wiring diagram of the primate visual system, dorsal and ventral streams from V1 through V2, V4, PIT and AIT to prefrontal cortex; in the middle the model, simple units (solid circles) and complex units (dashed circles) alternating from oriented S1 filters on a photograph of cows up to an animal versus non-animal classification unit, with bypass routes; on the right a table of the layers S1 to S4 and C1 to C3 with their receptive-field sizes from 0.2 to 7 degrees and their numbers of units from a million to a hundred
  • S and C alternate, V1 to IT. S units: template matching, a convolution, like a simple cell. C units: a max, like a complex cell, for tolerance to position and scale (next slide)
  • Hyperparameters (receptive-field sizes, tuning widths, pooling ranges) set by hand to match physiology, not fit to the task: "graduate student descent"
  • No backprop: higher S units learn by simple unsupervised, Hebbian-like rules, picking up frequent combinations of the C features below (from orientations to corners and curves); only the readout sees labels

Pooling over position, and over scale

HMAX's S1 Gabor filters in a grid: rows at 0, 45, 90 and 135 degrees; columns at sizes 7, 9, 11, 13, 35 and 37 pixels, each filter drawn at its true relative size. Magenta boxes group the sizes into scale bands 1 (7 and 9), 2 (11 and 13) and 8 (35 and 37), with an ellipsis for bands 3 to 7. Caption: C1 takes the max over two adjacent sizes and nearby positions
  • HMAX's S1 bank: Gabor filters at 4 orientations and 16 sizes. A C1 unit takes the max over nearby positions and over two adjacent sizes: 8 scale bands
  • So an object seen once at one size drives nearly the same C1 units at another size: HMAX has tolerance to scale built in
  • A network has no built-in scale pooling: its size tolerance comes from the sizes in its training photographs and from data augmentation, the rescaled copies of last lecture, and it fades outside the range it saw

Pooling: from a map to fewer numbers

A flower photograph and one of its C1 orientation maps, with an 8 by 8 area boxed. That area as a grid of its activation values, from 0.03 to 1, divided into 2 by 2 windows with the largest value in each window in bold. An arrow labeled 2 by 2 max, stride 2 leads to a 4 by 4 grid that holds only the winning values
  • A map says where a feature is, to the pixel; recognition needs whether, roughly where. Max-pooling keeps the largest value in each window, so a small shift of the feature gives the same answer. This is tolerance to position, as in a complex cell; HMAX's C stages do the same. Each filter's map is one channel

The same two steps, higher up

Wiring from C1 to C2. Bottom: three pairs of C1 units (orange), each pair a vertical and a horizontal unit at one position. Middle: three S2 units (green), each a weighted sum of the pair below it and tuned to a corner. Top: one C2 unit (orange), the max over the three S2 units, tuned to the corner at any of the three positions
  • A simple unit above V1 (HMAX's S2) is again a weighted sum, now over complex cells: it is selective for a conjunction of oriented edges at set positions, a corner, a curve, a textured patch
  • A complex unit above it (C2) pools those simple units over positions and sizes, so it responds to that conjunction anywhere in its field
  • Repeat a weighted sum, then pooling, and both selectivity and tolerance grow from V2 to V4 to IT

S2 and C2 on a real photograph

A black-eyed susan photograph passes through the HMAX stages from left to right. Four C1 maps, one per orientation, light up the petals that run in each direction. One S2 feature's template is tuned to 135-degree edges in its upper left and 45-degree edges along its bottom. Its S2 response map shows where that combination appears, with 6 by 6 pooling windows; pooling keeps one value per window, and C2, the maximum over all positions, is 0.689. A second row repeats this with the flower shifted 30 pixels up and 16 right: the S2 map hardly matches the first (correlation 0.04), yet C2 is 0.694, a change of 0.8 percent
  • Shift the flower: the S2 map moves, but C2, the max over positions, barely changes (0.689 → 0.694)

HMAX worked: predictions from V1 to behavior

Table from Serre and Poggio 2010: quantitative data compatible with the model, by area. Psychophysics: rapid animal categorization and the face inversion effect; LOC: face processing in fMRI; PFC: the differential role of IT and PFC in categorization; IT: tuning and invariance properties, read-out for object category, the average effect in IT; V4: the MAX operation, tuning for two-bar stimuli, two-spot interaction, tuning for boundary conformation, tuning for Cartesian and non-Cartesian gratings; V1: simple and complex cell tuning properties, the MAX operation in a subset of complex cells. Rows in blue are predictions the model made before the data, rows in red data consistent with the model, rows in black data used to set its parameters
  • One model, one set of parameters, tested from V1 to behavior
  • Some results were predicted before the data: max-like pooling in V4 and in some V1 complex cells
  • V4 tuning for curved boundaries (Pasupathy & Connor) was matched by fitting model units to each cell; a linear readout of the same model units named the category as well as a linear readout of recorded IT sites (Hung et al. 2005)

The test on behavior: rapid animal categorization

Performance in d-prime for the model, red solid line, and for human observers, blue dashed line, across four viewing distances, head, close-body, medium-body and far-body: both peak near 2.5 at close-body and fall to about 1.4 at far-body, with overlapping error bars The stimulus set: animals photographed at four viewing distances, head, close-body, medium-body and far-body, each with a matched distractor scene
  • Serre, Oliva & Poggio (2007): the animal / non-animal task of the vision lecture's opening movie. Photographs flashed for 20 ms, then a noise mask (a second image that overwrites the first) 50 ms after onset: too soon for feedback from higher areas to return
  • d′d': sensitivity, hits against false alarms, 0 = guessing. The model matched people at every viewing distance and was wrong on the same images more often than two equally good observers would be
  • Conclusion: with an early mask, one feedforward pass is enough

A mini-HMAX in PyTorch

class MiniHMAX(nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        self.s1 = nn.Sequential(nn.Conv2d(3, 16, kernel_size=5, padding=2), nn.ReLU())
        self.c1 = nn.MaxPool2d(4)                          # max over 4 × 4 blocks
        self.s2b = nn.Sequential(nn.Conv2d(16, 64, kernel_size=7, padding=3), nn.ReLU())
        self.c2b = nn.AdaptiveMaxPool2d(1)                 # max over all positions
        self.readout = nn.Linear(64, num_classes)

    def forward(self, images):                             # (batch, 3, 224, 224)
        hidden = self.s1(images)                           # (batch, 16, 224, 224)
        hidden = self.c1(hidden)                           # (batch, 16, 56, 56)
        hidden = self.s2b(hidden)                          # (batch, 64, 56, 56)
        pooled = self.c2b(hidden).flatten(1)               # (batch, 64)
        return self.readout(pooled)                        # class scores (logits)
  • S: template matching (convolution, ReLU); C: a maximum over positions. HMAX's bypass route (S2b: an S2 layer fed straight from C1), S1 → C1 → S2b → C2b. In the assignment: you train it with Adam (the training lecture's optimizer) and keep the epoch with the best validation accuracy

Then scale took over

ImageNet top-5 accuracy against year for five networks: AlexNet 2012, 83.6 percent with 8 layers; ZFNet 2013, 88.3 percent, 8 layers; GoogLeNet 2014, 93.3 percent, 22 layers; VGG-16 2014, 92.7 percent, 16 layers; ResNet-152 2015, 96.4 percent, 152 layers, highlighted
  • CNNs existed, LeCun's LeNet (1998), with every filter learned end to end by backprop; through the 2000s they and HMAX were about equally good, and neither got much better with more data
  • What changed in 2012: GPUs, and 1.2 million labeled photographs (ImageNet). AlexNet, then 8 → 152 layers in three years, better every year (top-5: the right label among the five best guesses)
  • AlexNet kept the same stack (a weighted sum, then a max) but learned every weight from the labels

Same operations, different settings and learning

  • Operations: the same. A weighted sum, then a pooling, stacked. HMAX's S and C; a network's convolution and max-pooling
  • Settings: HMAX took every filter size, pooling range and tuning width from physiology. A deep network sets them by an engineering criterion, whatever maximizes accuracy on held-out images, with filter sizes, strides and depth tried by hand
  • Learning: HMAX learned its S2 templates without labels and trained only a readout on top, animal or not. A network learns every weight by backpropagation from 1.2 million labels, 1,000 classes, supervised end to end
  • Scale tolerance: built into HMAX's C1; learned from augmentation in a network

AlexNet (2012): the first layer it learned

AlexNet's 96 first-layer filters from Krizhevsky et al. 2012, Figure 3: the top three rows mostly gray oriented edges and gratings at several sizes, the bottom three rows mostly color blobs and color edges
  • Krizhevsky, Sutskever & Hinton (2012): eight layers, 60 million weights, backprop on 1.2 million labeled photographs, split over two GPUs
  • What learning from labels produced at the first stage: 96 filters, 11 × 11 × 3: one GPU's half learned mostly color-free edges, the other's mostly color blobs. No one designed them: training on labels alone produced Gabor-like filters, like V1 simple cells

ResNet-50: four block groups

ResNet-50 from left to right: a 224 by 224 tiger image; conv1, a 7 by 7 convolution at stride 2, 112 by 112; max-pool, 56 by 56; layer1, 3 blocks, 56 by 56; layer2, 4 blocks, 28 by 28; layer3, 6 blocks, 14 by 14; layer4, 3 blocks, 7 by 7; global average, 1 by 1; readout
  • ResNet-50: conv1 and one max-pool, four block groups (layer1 to layer4), then global average pooling (each channel averaged to one number: the image's feature vector) and the readout
  • Each block group is a stack of residual blocks (a few convolutions whose input is added back to their output): 3 + 4 + 6 + 3 = 16 blocks, about 50 layers in all
  • Each group after the first halves the maps' height and width with a stride-2 convolution. The ResNet-18 of the previous assignment uses the same names

Freeze the encoder, or fine-tune the last block group

The same ResNet-50 diagram, now colored: conv1, max-pool and layer1 to layer3 in gray, frozen, weights kept; layer4 and the readout in magenta, trained on the new task, fine-tuned
  • The encoder is every layer before the readout. Frozen: weights fixed. Fine-tuning: keep training some pretrained weights
  • Middle road: freeze conv1–layer3, fine-tune layer4 and the readout. In the assignment: probe, then fine-tune
  • BatchNorm rescales each channel: by minibatch statistics in training mode, by stored averages in evaluation mode. model.eval() on new images; frozen layers in eval mode while fine-tuning

Adversarial examples

Goodfellow et al. 2015, Figure 1: a photograph of a panda, classified panda with 57.7 percent confidence, plus 0.007 times a colorful noise pattern, the sign of the gradient of the loss with respect to the pixels, equals an image that looks identical to the panda but is classified gibbon with 99.3 percent confidence
  • Nudge each pixel a tiny step along the gradient of the loss with respect to the pixels: "panda" becomes "gibbon", 99.3% confident; people see no change
  • Adversarial training trains on such images: the robust ResNet-50 (next slide)

Three ResNet-50s, one architecture, three trainings

The same ResNet-50 diagram as two slides ago: a tiger image, conv1, max-pool, four block groups layer1 to layer4, global average pooling and a readout; the architecture every one of the three networks shares
  • Supervised: ImageNet's 1.2 million labeled photographs; the network of the ResNet-50 slides
  • DINO (self-supervised): no labels; two augmented views of one photograph must agree
  • Robust: trained on adversarial examples (previous slide)
  • Same architecture, three sets of weights. In the assignment: saliency of the standard and robust networks

Recap

  • Simple and complex cells became convolution and max-pooling. The ventral stream: anatomy orders V1 → V2 → V4 → IT, cells prefer more complex things at each stage, responses start later at each stage
  • HMAX set the two operations from physiology and predicted data it was not fit to; networks then learned them from labels at scale
  • ResNet-50: residual blocks, BatchNorm (keep frozen layers in evaluation mode), fine-tune layer4; one architecture, three trainings (supervised, DINO, robust = trained on adversarial examples)
  • Adversarial examples: a tiny step of the pixels along the gradient of the loss changes the answer; the same gradient, read instead of stepped along, opens the next lecture

Further reading for the research trail

Optional: a few research-trail entries over the semester. Any paper mentioned in any lecture qualifies.

Minute paper

Submit on Canvas → Minute papers → Minute Paper 13 (access code read out in class).

Write three brief points in your own words:

  1. Name two of the three kinds of evidence that the ventral stream is organized as a hierarchy, and say in one sentence what each shows.
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Going further: more of the network

Optional reading: stride and padding, VGG, residual blocks, batch normalization, training and evaluation modes

Stride and padding set the size of a map

Two panels. Left: a 5 by 5 input surrounded by one ring of zeros, a 3 by 3 filter at its first position and, dashed, one step to the right, labeled stride 1; output 5 by 5. Right: the same input with the filter moved two steps, stride 2; output 3 by 3
  • Stride ss: how far the filter moves between positions; stride 2 halves the map, so a unit of the next layer reads a wider patch of the image: the receptive field grows, as from V1 to IT
  • Padding pp: rows and columns of zeros around the input, so the filter can also sit on the border
  • Output size for a k×kk \times k filter: Hl=⌊(Hl−1+2p−k)/s⌋+1H_l = \lfloor (H_{l-1} + 2p - k)/s \rfloor + 1. ResNet-50's first layer: 224, k=7k = 7, s=2s = 2, p=3p = 3, gives 112

VGG (2014): small filters, stacked

Left, one 7 by 7 filter: 49 weights, one ReLU. Right, three 3 by 3 filters in a row: the receptive field grows from 3 to 5 to 7 pixels, 27 weights, three ReLUs

  • Simonyan & Zisserman (2014) used only 3 × 3 filters, 16 to 19 layers deep. Three 3 × 3 filters in a row see the same 7 × 7 patch as one large filter, with 27 weights instead of 49 and two more nonlinearities
  • Deeper for less: 92.7% top-5 on ImageNet, and the first-layer filters still come out Gabor-like

Deeper networks: add the input back

A residual block: the block input A passes through a 1 by 1 convolution with BatchNorm and ReLU, a 3 by 3 convolution with BatchNorm and ReLU, and a 1 by 1 convolution with BatchNorm, together labeled F of A, the change the block learns; a green skip connection carries A unchanged around them to a plus node, followed by a ReLU, giving the block output g of A plus F of A
  • Stacking more plain layers made networks harder to train: a 56-layer network reached higher training error than a 20-layer one
  • A residual block computes a change F(A)\mathcal{F}(\boldsymbol{A}) and adds it to its input: output g(A+F(A))g(\boldsymbol{A} + \mathcal{F}(\boldsymbol{A})). The added path is a skip connection
  • If a block is not needed, its change can stay near zero and the input passes through: networks of 50 to 152 layers trained well
  • Each convolution is followed by BatchNorm, explained on its own slide

Normalization, from cortex to networks

Response of a model neuron against the contrast of its preferred grating, on a log axis, for three masks. Without a mask the response rises from 0 near 1 percent contrast to almost 1 at 100 percent; a weak and then a strong mask shift the curve to the right, so the same grating drives a weaker response. Above the plot: response equals drive divided by sigma plus the pooled drive of neighbors

Computed from the normalization model; schematic

  • Divisive normalization: a neuron's drive is divided by the pooled activity of its neighbors. A mask that drives the pool lowers the response at every contrast; found from the retina to cortex
  • HMAX's S1 divides each filter response by the energy of the image patch. AlexNet normalized across neighboring channels (local response normalization)
  • BatchNorm (2015, standard since ResNet) divides each channel by its spread over a minibatch: a simplified normalization, chosen because it speeds training

Batch normalization

Three histograms of one channel of ResNet-50's first layer over 64 photographs and every position. Before: mean minus 0.81, standard deviation 3.19, spread from about minus 10 to 10. After subtracting the batch mean and dividing by the batch standard deviation: mean 0, standard deviation 1. After the learned scale gamma and shift beta: mean 0.58, standard deviation 0.36

Computed: ResNet-50, 64 Imagenette photographs

  • As training changes the weights of earlier layers, the scale of each channel's values drifts. Batch normalization (BatchNorm) rescales every channel, layer after layer
  • For each channel: subtract the mean μB\mu_{\mathcal{B}} and divide by the standard deviation (s.d.) of the current minibatch (all its images and positions), then apply a learned scale γ\gamma and shift β\beta
  • z^=(z−μB) /σB2+ϵ,z~=γ z^+β\hat{z} = (z - \mu_{\mathcal{B}})\,/\sqrt{\sigma^2_{\mathcal{B}} + \epsilon}, \quad \tilde{z} = \gamma\,\hat{z} + \beta

Training mode and evaluation mode

Two rows. Training mode, model.train(): a minibatch of images gives the batch mean and variance, used to normalize before gamma and beta and the next layer; the batch statistics also update a stored running mean and variance. Evaluation mode, model.eval(): images to classify are normalized with the stored running mean and variance, then gamma and beta, then the next layer
  • Training mode normalizes with the minibatch's own statistics, and updates a running mean and variance: averages over the training batches
  • Evaluation mode (model.eval()) normalizes with those stored values instead
  • The stored values are not weights: no gradient changes them. model.eval() does not turn gradients off

- Last lecture's figure: the tiger in gray levels correlated with one 11 × 11 Gabor filter at every position, then a ReLU; the map is bright where the stripes run along the filter's angle - To carry forward: the network has one copy of the filter's weights, moved from position to position. Cortex has no way to move anything; the cortical-columns slide, after the complex cells, is what it does instead - Next slide: the same story in the brain, Hubel and Wiesel's simple and complex cells

- We stopped at the complex cell. Today: complex cells again, the visual areas above V1, and models of them; then how to see what a trained network uses - The picture is last lecture's summary of V1: center–surround inputs from the retina and LGN, simple cells built by summing rows of those inputs, complex cells built by pooling simple cells of one orientation at nearby positions - The minute paper asked which network operation each cell type corresponds to. Hubel and Wiesel described what the cells respond to, not the operation that builds them: a weighted sum fits the simple cell, and some pooling the complex cell. Which pooling, a sum or a max, is a model's choice; HMAX, later today, proposes the max, for tolerance to position and scale - Second half of today: explainability, the methods that show what a trained network's decisions depend on. The next lecture scores the same network against V4 and IT recordings

- The three frames: bar just left of the fields, only cell 1 fires a little; bar on cell 3's field, cell 3 and the complex cell fire; one field over, cell 4 fires instead, and the complex cell keeps firing - A complex cell pools several simple cells that prefer the same orientation and whose fields lie side by side. It keeps their orientation selectivity and gives up their position selectivity: a bar of the right orientation drives it wherever it falls within the pooled fields; a bar of the wrong orientation drives none of the simple cells, so the complex cell stays silent - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five

- A complex cell pools several simple cells that prefer the same orientation and whose fields lie side by side. It keeps their orientation selectivity and gives up their position selectivity: a bar of the right orientation drives it wherever it falls within the pooled fields; a bar of the wrong orientation drives none of the simple cells, so the complex cell stays silent - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five

- A complex cell pools several simple cells that prefer the same orientation and whose fields lie side by side. It keeps their orientation selectivity and gives up their position selectivity: a bar of the right orientation drives it wherever it falls within the pooled fields; a bar of the wrong orientation drives none of the simple cells, so the complex cell stays silent - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five - Tolerance is to position, not to orientation: the common slip in the minute papers. Turn the bar by 90° and the response of every cell in this picture goes to zero - Each simple cell's receptive field is the patch of the image it answers to; the five fields sit side by side - A convolutional layer is the simple cell's weighted sum applied at every position; pooling over a window is the complex cell. The Neocognitron and HMAX, later today, repeat the pair up the hierarchy - A sum and a max give the same tolerance in this picture; Lampl et al. (2004) found complex cells that behave like a max, the prediction marked blue on the HMAX table later today

- The ice-cube model (Hubel and Wiesel 1977): orientation columns run one way, ocular-dominance slabs the other, and a block about 1 mm across (a hypercolumn) holds every orientation for both eyes for one small patch of the visual field. The next block does the same for the next patch - The drawing is the textbook one (Kandel, Schwartz and Jessell, Principles of Neural Science). The colors code the preferred orientation; the blobs are cytochrome-oxidase patches, whose cells respond to color and are hardly orientation-selective - Replication: every position has its own cells with the same set of templates, each learned separately. A convolutional layer gets the same map with one copy of each filter applied everywhere, which is why it has so few weights (last lecture counted them). One proposal for why cortex's copies match: the same developmental rules and similar visual input at every position produce similar receptive fields - The drawing is a schematic. Real orientation maps are pinwheels (Bonhoeffer & Grinvald 1991), and mice have no orientation columns at all, yet their V1 still covers every orientation at every position

- The figure is last lecture's: DiCarlo, Zoccolan & Rust (2012), Fig. 1, the macaque ventral stream and its areas stacked with their typical response latencies and neuron counts - Latency: the time from the image appearing to the first spikes. Schmolesky et al. (1998) measured onset latency with the same stimuli in every area: LGN near 35 ms, V1 near 65, V2 near 80, V4 near 105; the figure's values are DiCarlo et al.'s rounded summary. The order is what matters: each stage starts later than the one below, so the first sweep up the stream is feedforward, which is what a feedforward model can hope to capture - Nothing says the hierarchy is a chain: every area also sends connections back down, and sideways; a later lecture returns to feedback

- Felleman and Van Essen (1991), Distributed hierarchical processing in the primate cerebral cortex: 32 areas, 305 connections, and the laminar rule that assigns each a direction. The diagram is the one every textbook reproduces - The rule: axons that start in the superficial layers (2 and 3) and end in layer 4 of the target are feedforward; axons that start in the deep layers and end outside layer 4 are feedback. Count every feedforward connection as "up" and the areas sort into levels - Many areas, many routes: the ventral stream of the previous slide is one path through this diagram, the one that ends in IT. The dorsal path through MT ends in the parietal areas

- Kobatake & Tanaka (1994) mapped receptive fields along the ventral stream: small near the fovea in V1, tens of degrees in anterior IT - This slide shows how far across the visual field each cell sees; the next slides show what the cells of each area prefer

- Gallant, Braun and Van Essen (1993), Selectivity for polar, hyperbolic and Cartesian gratings in macaque visual cortex; Gallant et al. (1996) extended it to V1, V2 and V4. The figure draws the three families from their formulas: Cartesian gratings vary along one direction, polar ones along the radius or the angle, hyperbolic ones along the product of the two coordinates - A V1 simple cell, a weighted sum over one oriented patch, prefers a Cartesian grating. A cell that prefers concentric rings must combine several orientations around a center: a second stage

- The table shows, for each area, examples of the critical features of cells that needed more than a bar or a spot - Kobatake and Tanaka (1994) found each cell's "critical feature" by starting from the object that drove it most and simplifying the picture step by step, keeping only the changes that did not lower the response. The table collects typical results by area - Desimone et al. (1984) had already found IT neurons that respond to faces, cartoon faces included, but not to scrambled parts or a hand; the face cells cluster in patches (Tsao et al. 2006) - The next two slides make V4 and V2 tuning precise: curvature (Pasupathy & Connor), then textures (Freeman et al.)

- Pasupathy and Connor (2001), Shape representation in area V4: position-specific tuning for boundary conformation. Each cell's responses to the 51 × 8 stimuli were fit by a Gaussian over two variables, the curvature along the boundary (concave to sharply convex) and its angular position relative to the shape's center - The shape panels are reproduced from Pospisil, Pasupathy and Bair (2018), Fig. 3, who reused the same stimulus set to test network units against V4 - Cadieu et al. (2007) fit HMAX's S2 units to these same V4 cells and matched their selectivity and their tolerance to shifts, the entry marked in red on the HMAX table later today

- Freeman, Ziemba, Heeger, Simoncelli and Movshon (2013), A functional and perceptual signature of the second visual area in primates and humans. The textures are synthesized by the Portilla–Simoncelli method, which starts from noise and adjusts it until a chosen set of statistics matches the original (panel d of their figure) - For the hierarchy: V2 reads the correlations between V1 filters, a second weighted-sum stage over V1's outputs, which is what the models later today build in

- THINGS (Hebart et al. 2019): 1,854 object concepts, 26,107 photographs, with human similarity judgments, fMRI, MEG and behavior collected on the same images - TVSD, the THINGS Ventral Stream Spiking Dataset (Papale et al. 2025): multi-unit activity from chronic high-channel-count Utah arrays in V1, V4 and IT of two monkeys, 22,248 photographs seen once and 100 seen thirty times; the assignment's model–brain part uses it, and the next lecture scores a network against it block by block - Triple-N (Li et al. 2026): Neuropixels recordings through macaque IT, including the middle and anterior category-selective regions, while the monkeys view scene photographs; the next lecture and the final projects come back to it - NSD, the Natural Scenes Dataset (Allen et al. 2022): eight participants, 30 to 40 sessions each, 7 T fMRI, about 73,000 photographs from COCO; the dataset behind most recent human model–brain comparisons - Recording methods: a tungsten microelectrode isolates one neuron at a time, the classic approach from Hubel and Wiesel to Pasupathy and Connor; a Utah array is a 4 × 4 mm grid of 96 to 100 electrodes implanted in cortex, recording multi-unit activity for months; a Neuropixels probe is a single thin shank with hundreds of recording sites, now used in monkeys too; fMRI records blood-oxygen signals in voxels a few millimetres across, over the whole brain, at the cost of time and spatial resolution - A site is one electrode and the few neurons nearest it

- So far: what the cells along the ventral stream prefer, how large their receptive fields are, and how fast they respond. Next: models that build these properties by stacking the simple and complex cell operations

- Fukushima built the Neocognitron directly from Hubel and Wiesel's two cell types: a plane of S-cells all sharing one template, and C-cells that pool over a neighborhood of S-cells in that plane. Stacking the pair gives position tolerance that grows with depth - It recognized handwritten digits whatever their position and with some distortion, in 1980, with unsupervised learning (a Hebbian rule that reinforced the strongest responses) and no backpropagation - The figure is the paper's Fig. 2: the cell planes and how each cell reads a small patch of the layer below; the paper's Fig. 1 maps Hubel and Wiesel's cell types onto these layers

- The figure is Serre, Oliva & Poggio (2007), Fig. 1: the wiring of the primate visual system (left) and the model (right); solid circles are simple units (tuning), dashed circles complex units (MAX); the table beside it gives each layer's receptive-field size in degrees and its number of units - HMAX's S1 units are Gabor filters at several scales; the max over position and scale gives C1; S2 units learn, without labels, the combinations of C1 features that occur often in natural images (in the 2007 implementation, by sampling patches of C1 responses); the only part trained on labels is the readout at the top - Receptive-field sizes, layer by layer, the tuning widths and the pooling were taken from recordings, so the model could be tested against data it was not fit to. The HMAX-worked slide is that test - I built this model during my PhD with Tomaso Poggio - No backprop was a choice: Crick (1989) argued that nothing in cortex could carry the exact error signals backpropagation needs, and most of us believed him. My own view has changed since: a large body of work now looks for biologically grounded approximations of backpropagation (feedback alignment, predictive coding), and the predictive-coding lecture takes it up

- The figure draws Gabor filters at four orientations and four sizes, as in HMAX's S1 layer (the full model uses 16 sizes grouped in 8 bands). The magenta boxes mark one C1 unit's pooling range in size; it also pools over a neighborhood of positions, as the complex cell did - Pooling over scale is what lets the model generalize from one training example at one size to the same object at another size, with no extra training. A convolutional network has no scale pooling: it learns size tolerance only from augmentation, the rescaled training copies of last lecture, and it breaks outside the sizes it was shown - Last lecture's four-filter demo showed the orientation half of this bank; the sizes are the other half

- The values are one C1 orientation map of a Flowers102 photograph, computed with the mini-HMAX S1 and C1 - Pooling has no weights to learn

- The diagram shows only the wiring from C1 to C2: S1 and C1 play V1's simple and complex cells; S2 units are weighted sums over patches of C1 responses, tuned to conjunctions of oriented edges; C2 units pool them; S3 and the readout sit at the top. Each step up doubles the size of the receptive field, roughly as from V1 to V2 to V4 to IT - The S2 unit's selectivity comes from the same mechanism as the simple cell's, a template matched against its inputs, except that the inputs are now complex cells, each already tolerant to position; the C2 unit's tolerance comes from the same max as the complex cell's. Two operations, repeated, give both the complexity and the tolerance the recordings showed, area by area - This is Hubel and Wiesel's own proposal, extended: Fukushima's Neocognitron and HMAX built it; the convolution and pooling layers of a deep network stack it the same way

- Computed with the mini-HMAX operations on a Flowers102 photograph from the assignment's data: fixed Gabor S1 (11 × 11, 4 orientations), C1 = max over 4 × 4 blocks, S2 = one feature, a 7 × 7 template (its tuning) over the four C1 maps; HMAX learns these templates from natural images without labels, C2 = max over all positions - The 6 × 6 local max in the middle is drawn only to show the pooling step; the network goes straight from S2 to the global max - The gold box on the photograph marks the region the strongest S2 unit sees; the gold ring on the S2 map marks that strongest response, and the gold square in the next panel the window it falls in - Correlation 0.04 between the two S2 maps, position by position: the map follows the flower. C2 changes by 0.8 percent: the answer does not

- Reading the table: rows are behavior (Psych., psychophysics), a human fMRI area (LOC), prefrontal cortex (PFC), then IT, V4, V1. Black: data used to set the parameters. Red: data the model agrees with. Blue: predictions made before the data - The table is from Serre and Poggio (2010), Communications of the ACM, with its own color code: black for data used to derive the parameters, red for data consistent with the model but not used to fit it, blue for correct predictions - The V4 MAX prediction (Riesenhuber and Poggio 1999) was tested by Gawne and Martin (2002); the V1 complex-cell MAX by Lampl, Ferster, Poggio and Riesenhuber (2004). Cadieu et al. (2007) fit the model's S2 units to Pasupathy and Connor's V4 cells tuned to curved boundaries - Hung et al. (2005): a linear readout of about 300 IT sites reports object category 100 ms after the image appears; the same readout works on the model's top units - HMAX was wrong in places. Because each part was constrained by data, those places could be found and tested

- d′ (d-prime) measures how well observers separate animal from non-animal images, correcting for how often they say "animal" overall; 0 is chance - The four distances go from a close-up of the head to a small animal in a wide scene; each animal image has a distractor scene matched for its statistics. Model (red) and observers (blue) agree across all four, and the images one finds hard the other tends to find hard too - The mask, shown 50 ms after the photograph appears (stimulus onset asynchrony) and lasting 80 ms, limits the time for feedback from higher areas, which is what makes the task a test of the feedforward sweep - Rapid categorization is the simplest piece of vision, not a representative one: attention, grouping and eye movements return when there is time, and so does feedback; a later lecture takes that up - Observers pressed animal / no animal; d′ was computed per viewing distance (head, close body, medium body, far body), and the model's readout was scored the same way on the same images

- A tensor is PyTorch's array: numbers in a grid with a shape. Images arrive as (batch, channels, height, width) - The bypass route (thin lines in the HMAX figure) goes from C1 straight to S2b and C2b, skipping the S2 to C3 layers of the main route - Keeping the epoch with the best validation accuracy is the generalization lecture's early stopping - s1 and s2b are learned here; in HMAX, S1 is a fixed bank of Gabor filters - Padding keeps the S maps the size of their input: (224 + 4 − 5) + 1 = 224 - out_channels is the number of filters and therefore of feature maps: 16, then 64 - Each S2b unit sees a 32 × 32 pixel patch of the image - Global max pooling keeps one number per map and ignores where the best match was - flatten(1) turns the (batch, 64, 1, 1) output of the pooling into (batch, 64) - Training uses the training loop of the backpropagation lecture: logits, cross-entropy, backward, step

- LeNet read handwritten digits on bank cheques in the 1990s, trained by backpropagation; its design is the Neocognitron's with learned filters. AlexNet is LeNet made deeper and wider, trained on two GPUs for a week - ImageNet: 1.2 million training photographs in 1,000 categories; top-5 accuracy counts an image right if the correct label is among the five most probable. The numbers on the figure are the ImageNet competition results of each year - In the mid-2000s, HMAX-style models (Serre, Wolf & Poggio 2005) and the convolutional networks of the time (Ranzato et al. 2007) were competitive with each other and with the rest of computer vision on small datasets such as Caltech-101. Once there were enough labels and enough compute for end-to-end training, hand-set constraints held networks back - ResNet-50, later today, is the modern version of this network; the assignment fine-tunes it - The accuracies on the figure are the ones commonly quoted; some are single networks, some ensembles of several

- Both families stack the two operations of V1 up a hierarchy; the difference is who sets the numbers and how the weights are learned - HMAX: S1 Gabors at sizes taken from V1 recordings; C1 and C2 pooling ranges from receptive-field sizes in V1, V4 and IT; S2 templates sampled from natural images with no labels; a support-vector machine or a linear readout trained on the top units for one binary task. AlexNet: the same kind of stack, every weight learned by gradient descent on the 1,000-way ImageNet task, with the filter sizes, strides and depth chosen by trial and error for accuracy - Backpropagation was the part the biology seemed to rule out (Crick 1989); the predictive-coding lecture returns to the approximations that have since been proposed

- In the figure, the top rows are the filters of one GPU (color-free edges), the bottom rows those of the other (color blobs) - Reproduced from Krizhevsky et al. (2012), Fig. 3. Their network ran on two GPUs; the top three rows are the filters learned on one GPU (mostly color-free edges), the bottom three on the other (mostly color) - The color blobs in the lower rows recall the color-selective cells of V1's blobs; the record of one unit at a time is the electrophysiologist's method, applied to a network - How to show a filter: a k × k × 3 filter is a tiny RGB image once its values are rescaled to the display range, with zero mapped to gray (a weight of zero). Each filter is usually scaled on its own, so compare patterns, not brightness - The network was trained only to name ImageNet categories; nothing in its objective mentions edges or V1 - Sparse coding of natural images (Olshausen & Field 1996) reaches similar oriented filters from unlabeled patches; here they come from a network trained with labels. Two very different objectives, one kind of first stage

- Two ways to shrink a map: pooling, or a stride-2 convolution; many modern networks use mostly the second, then one global average. Lin, Chen & Yan (2014), Network in Network, introduced global average pooling in place of large dense layers at the top of a network - In PyTorch: nn.MaxPool2d(2) for max-pooling; nn.AdaptiveAvgPool2d(1) averages each channel down to a single value - A residual block adds its input back to its output, which is what lets 50 layers train (slides after the recap) - Map sizes are those of torchvision's ResNet-50 on a 224 × 224 input; the boxes are drawn smaller as the maps shrink, not to scale - "50" counts the layers with weights: conv1, three convolutions in each of the 16 blocks, and the readout: 1 + 48 + 1 - ResNet-18 uses two-convolution blocks, two per group

- Same diagram as on "ResNet-50: four block groups"; gray parts keep their ImageNet weights, magenta parts change - BatchNorm rescales each channel by the mean and variance of the current minibatch in training mode, and by stored running averages in evaluation mode. Freezing weights does not freeze those running averages: in model.train() they keep changing, even for frozen layers, and a network run on new images in training mode is normalized by whatever batch it sees. Two consequences: evaluate pretrained networks in model.eval(); when fine-tuning, put the frozen layers' BatchNorm back in eval mode after every model.train(). The provided code does this; the slides after the recap show the mechanism - Encoder: the part of a network that maps an image to its feature vector; here, everything up to the global average pooling. The generative-models lecture uses the same word for the first half of an autoencoder - Kumar et al. (2022): fine-tuning every layer did slightly better on images like the training images and worse on shifted ones - The backpropagation lecture and the second assignment compared the two extremes: a new readout on frozen features, or backprop into every layer - In PyTorch: call requires_grad_(False) on the parameters that stay frozen, and give the optimizer only the parameters that should change. Then call .eval() on the frozen BatchNorm modules after every model.train(): their running mean and variance are not parameters, so requires_grad_(False) does not stop them from updating - The assignment fine-tunes this way, with gradient descent and momentum, and shrinks the learning rate along a cosine curve to zero (CosineAnnealingLR), a smoother version of the training lecture's shrinking step - For brains, the same question: after an animal learns a new task, has the representation changed, or only the readout from it?

- The fast gradient sign method: x + ε · sign(∇ₓ loss), with ε = 0.007 of the pixel range. The same gradient with respect to the pixels is the saliency map of the next lecture, used here to change the answer rather than to explain it - Adversarial training: at each step, perturb the minibatch to maximize the loss within a small budget, then train on the perturbed images. It costs accuracy (a few points on ImageNet) and buys robustness to these perturbations - Robust networks' gradients look more like the object, and their representations predict some neural data better; the next lecture asks whether the robust ResNet-50 predicts IT better than the standard one - People are not fooled by perturbations this small; with a very brief presentation, larger perturbations made for many networks at once can bias human choices slightly (Elsayed et al. 2018)

- The three networks share torchvision's ResNet-50 architecture: the same 16 residual blocks in four groups, the same 2,048-number feature vector. Only the weights differ, because the training objective differs - DINO (Caron et al. 2021): a student network sees one augmented crop of a photograph and must match a teacher network's output on another crop of the same photograph; no label is ever used. The second assignment's DINO ResNet-50 is this network - You fit the same probe on all three frozen encoders and fine-tune `layer4` (the freeze slide); the next lecture asks which of the three looks most like IT

- The slides after the minute paper are optional reading on the network: stride and padding, VGG, residual blocks, BatchNorm, and training versus evaluation mode - Next lecture: which pixels one decision used (attribution maps, and how to test one), what a single unit responds to (feature visualization), then the same network scored against the ventral stream

- Colin, Fel, Cadène & Serre (2022), What I cannot predict, I do not understand: a human-centered evaluation framework for explainability methods - Hermann, Mobahi, Fel & Mozer (2024), On the foundations of shortcut learning - DeGrave, Janizek & Lee (2021), AI for radiographic COVID-19 detection selects shortcuts over signal - Zimmermann, Klein & Brendel (2023), Scale alone does not improve mechanistic interpretability in vision models - Kantamneni, Engels, Rajamanoharan, Tegmark & Nanda (2025), Are sparse autoencoders useful? A case study in sparse probing - Fel et al. (2023), Unlocking feature visualization for deep networks with magnitude constrained optimization - Also worth reading: Adebayo et al. (2018) on sanity checks for maps, Bau et al. (2017) on Network Dissection - Before posting, skim the existing entries on your paper; your post must add something not already said

- Submitted on Canvas (Minute papers → Minute Paper 13); credit for a thoughtful attempt

- One filter at every position gives a map as large as the image; stride and padding change that - ⌊ ⌋ rounds down to a whole number - Padding that keeps the size unchanged at stride 1 (p = (k − 1)/2 for odd k) is often called "same"; no padding is "valid" - Downsampling, by stride or by pooling, makes each unit of a later layer depend on a larger region of the image: its receptive field grows, as it does from V1 to IT - In PyTorch: nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3) is ResNet-50's first layer

- VGG, from the Visual Geometry Group at Oxford: the whole network is 3 × 3 convolutions, 2 × 2 max-pooling and three dense layers at the top; VGG-16 has 138 million weights, most of them in the dense layers - The receptive-field arithmetic is the stride-and-padding slide's: each 3 × 3 layer adds two pixels to the field, so three of them cover seven - The residual-block slide is the step that made networks of 50 to 152 layers trainable

- He, Zhang, Ren & Sun (2016), Fig. 1 (CIFAR-10) and Fig. 4 (ImageNet): plain networks got worse on their own training images as layers were added. That is a training problem, not overfitting - The block in the figure is ResNet-50's "bottleneck" block (He et al. 2016, Fig. 5 right): a 1 × 1 convolution reduces the number of channels, the 3 × 3 convolution works on fewer channels, a second 1 × 1 convolution restores them. A 1 × 1 convolution is a weighted sum across channels at each position - During backprop, the skip connection also carries the error signal straight to earlier blocks - A block's input and output must have the same shape to be added; where a block changes the size or the number of channels, the skip path gets its own 1 × 1 convolution

- Carandini and Heeger called divisive normalization a canonical neural computation: it explains contrast saturation, cross-orientation suppression and surround suppression in V1, and appears in olfaction, audition, attention and multisensory integration - In biology the pool is other neurons responding to the same image; in BatchNorm it is the same channel across other images in the minibatch and across positions, which is easy to compute on a GPU but has no counterpart in a brain that sees one image at a time - VGG dropped AlexNet's local response normalization because it did not help accuracy; BatchNorm came back as an engineering fix for training speed, not as a model of cortex

- Computed here: one channel of ResNet-50's first convolution, before its BatchNorm, on 64 Imagenette validation photographs (a 10-class subset of ImageNet), at every one of the 112 × 112 positions. Batch mean −0.81, batch variance 10.18 - The standard deviation is the square root of the variance σ²_B. ε is a small constant (1e-5 in PyTorch) that avoids dividing by zero - γ and β are learned weights, one pair per channel; they let the network undo the normalization where that helps - Ioffe & Szegedy (2015) introduced BatchNorm; with it, networks trained in far fewer steps. Their reason was the drift above; later work (Santurkar et al. 2018) found BatchNorm helps mostly by making the loss surface smoother, so larger learning rates work. Either way, the practical consequence is the train/eval difference on the next slide - A neuroscientist's question: is this divisive normalization (Carandini & Heeger 2012)? Not quite. BatchNorm divides by the spread across the images of a batch and freezes those numbers at test time; cortex divides each neuron's response by its neighbors' responses to the same image

- The figure is a schematic of one BatchNorm layer - The running mean and variance are buffers: tensors saved with the model, updated by each forward pass in training mode, read in evaluation mode. For the channel on the previous slide, ResNet-50 stores a running mean of 0.09 and a running variance of 13.84 - model.train() and model.eval() switch BatchNorm (and dropout) between the two behaviors. Whether gradients are computed is set separately, by requires_grad and torch.no_grad - PyTorch's running statistics are exponential moving averages: each training batch moves them 10% of the way toward that batch's values (momentum 0.1; this "momentum" is a BatchNorm setting, unrelated to the training lecture's optimizer momentum)