model.eval()) normalizes with those stored values insteadmodel.eval() does not turn gradients off- Last lecture's figure: the tiger in gray levels correlated with one 11 × 11 Gabor filter at every position, then a ReLU; the map is bright where the stripes run along the filter's angle - To carry forward: the network has one copy of the filter's weights, moved from position to position. Cortex has no way to move anything; the cortical-columns slide, after the complex cells, is what it does instead - Next slide: the same story in the brain, Hubel and Wiesel's simple and complex cells
- We stopped at the complex cell. Today: complex cells again, the visual areas above V1, and models of them; then how to see what a trained network uses - The picture is last lecture's summary of V1: center–surround inputs from the retina and LGN, simple cells built by summing rows of those inputs, complex cells built by pooling simple cells of one orientation at nearby positions - The minute paper asked which network operation each cell type corresponds to. Hubel and Wiesel described what the cells respond to, not the operation that builds them: a weighted sum fits the simple cell, and some pooling the complex cell. Which pooling, a sum or a max, is a model's choice; HMAX, later today, proposes the max, for tolerance to position and scale - Second half of today: explainability, the methods that show what a trained network's decisions depend on. The next lecture scores the same network against V4 and IT recordings
- The three frames: bar just left of the fields, only cell 1 fires a little; bar on cell 3's field, cell 3 and the complex cell fire; one field over, cell 4 fires instead, and the complex cell keeps firing - A complex cell pools several simple cells that prefer the same orientation and whose fields lie side by side. It keeps their orientation selectivity and gives up their position selectivity: a bar of the right orientation drives it wherever it falls within the pooled fields; a bar of the wrong orientation drives none of the simple cells, so the complex cell stays silent - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five
- A complex cell pools several simple cells that prefer the same orientation and whose fields lie side by side. It keeps their orientation selectivity and gives up their position selectivity: a bar of the right orientation drives it wherever it falls within the pooled fields; a bar of the wrong orientation drives none of the simple cells, so the complex cell stays silent - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five
- A complex cell pools several simple cells that prefer the same orientation and whose fields lie side by side. It keeps their orientation selectivity and gives up their position selectivity: a bar of the right orientation drives it wherever it falls within the pooled fields; a bar of the wrong orientation drives none of the simple cells, so the complex cell stays silent - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five - Tolerance is to position, not to orientation: the common slip in the minute papers. Turn the bar by 90° and the response of every cell in this picture goes to zero - Each simple cell's receptive field is the patch of the image it answers to; the five fields sit side by side - A convolutional layer is the simple cell's weighted sum applied at every position; pooling over a window is the complex cell. The Neocognitron and HMAX, later today, repeat the pair up the hierarchy - A sum and a max give the same tolerance in this picture; Lampl et al. (2004) found complex cells that behave like a max, the prediction marked blue on the HMAX table later today
- The ice-cube model (Hubel and Wiesel 1977): orientation columns run one way, ocular-dominance slabs the other, and a block about 1 mm across (a hypercolumn) holds every orientation for both eyes for one small patch of the visual field. The next block does the same for the next patch - The drawing is the textbook one (Kandel, Schwartz and Jessell, Principles of Neural Science). The colors code the preferred orientation; the blobs are cytochrome-oxidase patches, whose cells respond to color and are hardly orientation-selective - Replication: every position has its own cells with the same set of templates, each learned separately. A convolutional layer gets the same map with one copy of each filter applied everywhere, which is why it has so few weights (last lecture counted them). One proposal for why cortex's copies match: the same developmental rules and similar visual input at every position produce similar receptive fields - The drawing is a schematic. Real orientation maps are pinwheels (Bonhoeffer & Grinvald 1991), and mice have no orientation columns at all, yet their V1 still covers every orientation at every position
- The figure is last lecture's: DiCarlo, Zoccolan & Rust (2012), Fig. 1, the macaque ventral stream and its areas stacked with their typical response latencies and neuron counts - Latency: the time from the image appearing to the first spikes. Schmolesky et al. (1998) measured onset latency with the same stimuli in every area: LGN near 35 ms, V1 near 65, V2 near 80, V4 near 105; the figure's values are DiCarlo et al.'s rounded summary. The order is what matters: each stage starts later than the one below, so the first sweep up the stream is feedforward, which is what a feedforward model can hope to capture - Nothing says the hierarchy is a chain: every area also sends connections back down, and sideways; a later lecture returns to feedback
- Felleman and Van Essen (1991), Distributed hierarchical processing in the primate cerebral cortex: 32 areas, 305 connections, and the laminar rule that assigns each a direction. The diagram is the one every textbook reproduces - The rule: axons that start in the superficial layers (2 and 3) and end in layer 4 of the target are feedforward; axons that start in the deep layers and end outside layer 4 are feedback. Count every feedforward connection as "up" and the areas sort into levels - Many areas, many routes: the ventral stream of the previous slide is one path through this diagram, the one that ends in IT. The dorsal path through MT ends in the parietal areas
- Kobatake & Tanaka (1994) mapped receptive fields along the ventral stream: small near the fovea in V1, tens of degrees in anterior IT - This slide shows how far across the visual field each cell sees; the next slides show what the cells of each area prefer
- Gallant, Braun and Van Essen (1993), Selectivity for polar, hyperbolic and Cartesian gratings in macaque visual cortex; Gallant et al. (1996) extended it to V1, V2 and V4. The figure draws the three families from their formulas: Cartesian gratings vary along one direction, polar ones along the radius or the angle, hyperbolic ones along the product of the two coordinates - A V1 simple cell, a weighted sum over one oriented patch, prefers a Cartesian grating. A cell that prefers concentric rings must combine several orientations around a center: a second stage
- The table shows, for each area, examples of the critical features of cells that needed more than a bar or a spot - Kobatake and Tanaka (1994) found each cell's "critical feature" by starting from the object that drove it most and simplifying the picture step by step, keeping only the changes that did not lower the response. The table collects typical results by area - Desimone et al. (1984) had already found IT neurons that respond to faces, cartoon faces included, but not to scrambled parts or a hand; the face cells cluster in patches (Tsao et al. 2006) - The next two slides make V4 and V2 tuning precise: curvature (Pasupathy & Connor), then textures (Freeman et al.)
- Pasupathy and Connor (2001), Shape representation in area V4: position-specific tuning for boundary conformation. Each cell's responses to the 51 × 8 stimuli were fit by a Gaussian over two variables, the curvature along the boundary (concave to sharply convex) and its angular position relative to the shape's center - The shape panels are reproduced from Pospisil, Pasupathy and Bair (2018), Fig. 3, who reused the same stimulus set to test network units against V4 - Cadieu et al. (2007) fit HMAX's S2 units to these same V4 cells and matched their selectivity and their tolerance to shifts, the entry marked in red on the HMAX table later today
- Freeman, Ziemba, Heeger, Simoncelli and Movshon (2013), A functional and perceptual signature of the second visual area in primates and humans. The textures are synthesized by the Portilla–Simoncelli method, which starts from noise and adjusts it until a chosen set of statistics matches the original (panel d of their figure) - For the hierarchy: V2 reads the correlations between V1 filters, a second weighted-sum stage over V1's outputs, which is what the models later today build in
- THINGS (Hebart et al. 2019): 1,854 object concepts, 26,107 photographs, with human similarity judgments, fMRI, MEG and behavior collected on the same images - TVSD, the THINGS Ventral Stream Spiking Dataset (Papale et al. 2025): multi-unit activity from chronic high-channel-count Utah arrays in V1, V4 and IT of two monkeys, 22,248 photographs seen once and 100 seen thirty times; the assignment's model–brain part uses it, and the next lecture scores a network against it block by block - Triple-N (Li et al. 2026): Neuropixels recordings through macaque IT, including the middle and anterior category-selective regions, while the monkeys view scene photographs; the next lecture and the final projects come back to it - NSD, the Natural Scenes Dataset (Allen et al. 2022): eight participants, 30 to 40 sessions each, 7 T fMRI, about 73,000 photographs from COCO; the dataset behind most recent human model–brain comparisons - Recording methods: a tungsten microelectrode isolates one neuron at a time, the classic approach from Hubel and Wiesel to Pasupathy and Connor; a Utah array is a 4 × 4 mm grid of 96 to 100 electrodes implanted in cortex, recording multi-unit activity for months; a Neuropixels probe is a single thin shank with hundreds of recording sites, now used in monkeys too; fMRI records blood-oxygen signals in voxels a few millimetres across, over the whole brain, at the cost of time and spatial resolution - A site is one electrode and the few neurons nearest it
- So far: what the cells along the ventral stream prefer, how large their receptive fields are, and how fast they respond. Next: models that build these properties by stacking the simple and complex cell operations
- Fukushima built the Neocognitron directly from Hubel and Wiesel's two cell types: a plane of S-cells all sharing one template, and C-cells that pool over a neighborhood of S-cells in that plane. Stacking the pair gives position tolerance that grows with depth - It recognized handwritten digits whatever their position and with some distortion, in 1980, with unsupervised learning (a Hebbian rule that reinforced the strongest responses) and no backpropagation - The figure is the paper's Fig. 2: the cell planes and how each cell reads a small patch of the layer below; the paper's Fig. 1 maps Hubel and Wiesel's cell types onto these layers
- The figure is Serre, Oliva & Poggio (2007), Fig. 1: the wiring of the primate visual system (left) and the model (right); solid circles are simple units (tuning), dashed circles complex units (MAX); the table beside it gives each layer's receptive-field size in degrees and its number of units - HMAX's S1 units are Gabor filters at several scales; the max over position and scale gives C1; S2 units learn, without labels, the combinations of C1 features that occur often in natural images (in the 2007 implementation, by sampling patches of C1 responses); the only part trained on labels is the readout at the top - Receptive-field sizes, layer by layer, the tuning widths and the pooling were taken from recordings, so the model could be tested against data it was not fit to. The HMAX-worked slide is that test - I built this model during my PhD with Tomaso Poggio - No backprop was a choice: Crick (1989) argued that nothing in cortex could carry the exact error signals backpropagation needs, and most of us believed him. My own view has changed since: a large body of work now looks for biologically grounded approximations of backpropagation (feedback alignment, predictive coding), and the predictive-coding lecture takes it up
- The figure draws Gabor filters at four orientations and four sizes, as in HMAX's S1 layer (the full model uses 16 sizes grouped in 8 bands). The magenta boxes mark one C1 unit's pooling range in size; it also pools over a neighborhood of positions, as the complex cell did - Pooling over scale is what lets the model generalize from one training example at one size to the same object at another size, with no extra training. A convolutional network has no scale pooling: it learns size tolerance only from augmentation, the rescaled training copies of last lecture, and it breaks outside the sizes it was shown - Last lecture's four-filter demo showed the orientation half of this bank; the sizes are the other half
- The values are one C1 orientation map of a Flowers102 photograph, computed with the mini-HMAX S1 and C1 - Pooling has no weights to learn
- The diagram shows only the wiring from C1 to C2: S1 and C1 play V1's simple and complex cells; S2 units are weighted sums over patches of C1 responses, tuned to conjunctions of oriented edges; C2 units pool them; S3 and the readout sit at the top. Each step up doubles the size of the receptive field, roughly as from V1 to V2 to V4 to IT - The S2 unit's selectivity comes from the same mechanism as the simple cell's, a template matched against its inputs, except that the inputs are now complex cells, each already tolerant to position; the C2 unit's tolerance comes from the same max as the complex cell's. Two operations, repeated, give both the complexity and the tolerance the recordings showed, area by area - This is Hubel and Wiesel's own proposal, extended: Fukushima's Neocognitron and HMAX built it; the convolution and pooling layers of a deep network stack it the same way
- Computed with the mini-HMAX operations on a Flowers102 photograph from the assignment's data: fixed Gabor S1 (11 × 11, 4 orientations), C1 = max over 4 × 4 blocks, S2 = one feature, a 7 × 7 template (its tuning) over the four C1 maps; HMAX learns these templates from natural images without labels, C2 = max over all positions - The 6 × 6 local max in the middle is drawn only to show the pooling step; the network goes straight from S2 to the global max - The gold box on the photograph marks the region the strongest S2 unit sees; the gold ring on the S2 map marks that strongest response, and the gold square in the next panel the window it falls in - Correlation 0.04 between the two S2 maps, position by position: the map follows the flower. C2 changes by 0.8 percent: the answer does not
- Reading the table: rows are behavior (Psych., psychophysics), a human fMRI area (LOC), prefrontal cortex (PFC), then IT, V4, V1. Black: data used to set the parameters. Red: data the model agrees with. Blue: predictions made before the data - The table is from Serre and Poggio (2010), Communications of the ACM, with its own color code: black for data used to derive the parameters, red for data consistent with the model but not used to fit it, blue for correct predictions - The V4 MAX prediction (Riesenhuber and Poggio 1999) was tested by Gawne and Martin (2002); the V1 complex-cell MAX by Lampl, Ferster, Poggio and Riesenhuber (2004). Cadieu et al. (2007) fit the model's S2 units to Pasupathy and Connor's V4 cells tuned to curved boundaries - Hung et al. (2005): a linear readout of about 300 IT sites reports object category 100 ms after the image appears; the same readout works on the model's top units - HMAX was wrong in places. Because each part was constrained by data, those places could be found and tested
- d′ (d-prime) measures how well observers separate animal from non-animal images, correcting for how often they say "animal" overall; 0 is chance - The four distances go from a close-up of the head to a small animal in a wide scene; each animal image has a distractor scene matched for its statistics. Model (red) and observers (blue) agree across all four, and the images one finds hard the other tends to find hard too - The mask, shown 50 ms after the photograph appears (stimulus onset asynchrony) and lasting 80 ms, limits the time for feedback from higher areas, which is what makes the task a test of the feedforward sweep - Rapid categorization is the simplest piece of vision, not a representative one: attention, grouping and eye movements return when there is time, and so does feedback; a later lecture takes that up - Observers pressed animal / no animal; d′ was computed per viewing distance (head, close body, medium body, far body), and the model's readout was scored the same way on the same images
- A tensor is PyTorch's array: numbers in a grid with a shape. Images arrive as (batch, channels, height, width) - The bypass route (thin lines in the HMAX figure) goes from C1 straight to S2b and C2b, skipping the S2 to C3 layers of the main route - Keeping the epoch with the best validation accuracy is the generalization lecture's early stopping - s1 and s2b are learned here; in HMAX, S1 is a fixed bank of Gabor filters - Padding keeps the S maps the size of their input: (224 + 4 − 5) + 1 = 224 - out_channels is the number of filters and therefore of feature maps: 16, then 64 - Each S2b unit sees a 32 × 32 pixel patch of the image - Global max pooling keeps one number per map and ignores where the best match was - flatten(1) turns the (batch, 64, 1, 1) output of the pooling into (batch, 64) - Training uses the training loop of the backpropagation lecture: logits, cross-entropy, backward, step
- LeNet read handwritten digits on bank cheques in the 1990s, trained by backpropagation; its design is the Neocognitron's with learned filters. AlexNet is LeNet made deeper and wider, trained on two GPUs for a week - ImageNet: 1.2 million training photographs in 1,000 categories; top-5 accuracy counts an image right if the correct label is among the five most probable. The numbers on the figure are the ImageNet competition results of each year - In the mid-2000s, HMAX-style models (Serre, Wolf & Poggio 2005) and the convolutional networks of the time (Ranzato et al. 2007) were competitive with each other and with the rest of computer vision on small datasets such as Caltech-101. Once there were enough labels and enough compute for end-to-end training, hand-set constraints held networks back - ResNet-50, later today, is the modern version of this network; the assignment fine-tunes it - The accuracies on the figure are the ones commonly quoted; some are single networks, some ensembles of several
- Both families stack the two operations of V1 up a hierarchy; the difference is who sets the numbers and how the weights are learned - HMAX: S1 Gabors at sizes taken from V1 recordings; C1 and C2 pooling ranges from receptive-field sizes in V1, V4 and IT; S2 templates sampled from natural images with no labels; a support-vector machine or a linear readout trained on the top units for one binary task. AlexNet: the same kind of stack, every weight learned by gradient descent on the 1,000-way ImageNet task, with the filter sizes, strides and depth chosen by trial and error for accuracy - Backpropagation was the part the biology seemed to rule out (Crick 1989); the predictive-coding lecture returns to the approximations that have since been proposed
- In the figure, the top rows are the filters of one GPU (color-free edges), the bottom rows those of the other (color blobs) - Reproduced from Krizhevsky et al. (2012), Fig. 3. Their network ran on two GPUs; the top three rows are the filters learned on one GPU (mostly color-free edges), the bottom three on the other (mostly color) - The color blobs in the lower rows recall the color-selective cells of V1's blobs; the record of one unit at a time is the electrophysiologist's method, applied to a network - How to show a filter: a k × k × 3 filter is a tiny RGB image once its values are rescaled to the display range, with zero mapped to gray (a weight of zero). Each filter is usually scaled on its own, so compare patterns, not brightness - The network was trained only to name ImageNet categories; nothing in its objective mentions edges or V1 - Sparse coding of natural images (Olshausen & Field 1996) reaches similar oriented filters from unlabeled patches; here they come from a network trained with labels. Two very different objectives, one kind of first stage
- Two ways to shrink a map: pooling, or a stride-2 convolution; many modern networks use mostly the second, then one global average. Lin, Chen & Yan (2014), Network in Network, introduced global average pooling in place of large dense layers at the top of a network - In PyTorch: nn.MaxPool2d(2) for max-pooling; nn.AdaptiveAvgPool2d(1) averages each channel down to a single value - A residual block adds its input back to its output, which is what lets 50 layers train (slides after the recap) - Map sizes are those of torchvision's ResNet-50 on a 224 × 224 input; the boxes are drawn smaller as the maps shrink, not to scale - "50" counts the layers with weights: conv1, three convolutions in each of the 16 blocks, and the readout: 1 + 48 + 1 - ResNet-18 uses two-convolution blocks, two per group
- Same diagram as on "ResNet-50: four block groups"; gray parts keep their ImageNet weights, magenta parts change - BatchNorm rescales each channel by the mean and variance of the current minibatch in training mode, and by stored running averages in evaluation mode. Freezing weights does not freeze those running averages: in model.train() they keep changing, even for frozen layers, and a network run on new images in training mode is normalized by whatever batch it sees. Two consequences: evaluate pretrained networks in model.eval(); when fine-tuning, put the frozen layers' BatchNorm back in eval mode after every model.train(). The provided code does this; the slides after the recap show the mechanism - Encoder: the part of a network that maps an image to its feature vector; here, everything up to the global average pooling. The generative-models lecture uses the same word for the first half of an autoencoder - Kumar et al. (2022): fine-tuning every layer did slightly better on images like the training images and worse on shifted ones - The backpropagation lecture and the second assignment compared the two extremes: a new readout on frozen features, or backprop into every layer - In PyTorch: call requires_grad_(False) on the parameters that stay frozen, and give the optimizer only the parameters that should change. Then call .eval() on the frozen BatchNorm modules after every model.train(): their running mean and variance are not parameters, so requires_grad_(False) does not stop them from updating - The assignment fine-tunes this way, with gradient descent and momentum, and shrinks the learning rate along a cosine curve to zero (CosineAnnealingLR), a smoother version of the training lecture's shrinking step - For brains, the same question: after an animal learns a new task, has the representation changed, or only the readout from it?
- The fast gradient sign method: x + ε · sign(∇ₓ loss), with ε = 0.007 of the pixel range. The same gradient with respect to the pixels is the saliency map of the next lecture, used here to change the answer rather than to explain it - Adversarial training: at each step, perturb the minibatch to maximize the loss within a small budget, then train on the perturbed images. It costs accuracy (a few points on ImageNet) and buys robustness to these perturbations - Robust networks' gradients look more like the object, and their representations predict some neural data better; the next lecture asks whether the robust ResNet-50 predicts IT better than the standard one - People are not fooled by perturbations this small; with a very brief presentation, larger perturbations made for many networks at once can bias human choices slightly (Elsayed et al. 2018)
- The three networks share torchvision's ResNet-50 architecture: the same 16 residual blocks in four groups, the same 2,048-number feature vector. Only the weights differ, because the training objective differs - DINO (Caron et al. 2021): a student network sees one augmented crop of a photograph and must match a teacher network's output on another crop of the same photograph; no label is ever used. The second assignment's DINO ResNet-50 is this network - You fit the same probe on all three frozen encoders and fine-tune `layer4` (the freeze slide); the next lecture asks which of the three looks most like IT
- The slides after the minute paper are optional reading on the network: stride and padding, VGG, residual blocks, BatchNorm, and training versus evaluation mode - Next lecture: which pixels one decision used (attribution maps, and how to test one), what a single unit responds to (feature visualization), then the same network scored against the ventral stream
- Colin, Fel, Cadène & Serre (2022), What I cannot predict, I do not understand: a human-centered evaluation framework for explainability methods - Hermann, Mobahi, Fel & Mozer (2024), On the foundations of shortcut learning - DeGrave, Janizek & Lee (2021), AI for radiographic COVID-19 detection selects shortcuts over signal - Zimmermann, Klein & Brendel (2023), Scale alone does not improve mechanistic interpretability in vision models - Kantamneni, Engels, Rajamanoharan, Tegmark & Nanda (2025), Are sparse autoencoders useful? A case study in sparse probing - Fel et al. (2023), Unlocking feature visualization for deep networks with magnitude constrained optimization - Also worth reading: Adebayo et al. (2018) on sanity checks for maps, Bau et al. (2017) on Network Dissection - Before posting, skim the existing entries on your paper; your post must add something not already said
- Submitted on Canvas (Minute papers → Minute Paper 13); credit for a thoughtful attempt
- One filter at every position gives a map as large as the image; stride and padding change that - ⌊ ⌋ rounds down to a whole number - Padding that keeps the size unchanged at stride 1 (p = (k − 1)/2 for odd k) is often called "same"; no padding is "valid" - Downsampling, by stride or by pooling, makes each unit of a later layer depend on a larger region of the image: its receptive field grows, as it does from V1 to IT - In PyTorch: nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3) is ResNet-50's first layer
- VGG, from the Visual Geometry Group at Oxford: the whole network is 3 × 3 convolutions, 2 × 2 max-pooling and three dense layers at the top; VGG-16 has 138 million weights, most of them in the dense layers - The receptive-field arithmetic is the stride-and-padding slide's: each 3 × 3 layer adds two pixels to the field, so three of them cover seven - The residual-block slide is the step that made networks of 50 to 152 layers trainable
- He, Zhang, Ren & Sun (2016), Fig. 1 (CIFAR-10) and Fig. 4 (ImageNet): plain networks got worse on their own training images as layers were added. That is a training problem, not overfitting - The block in the figure is ResNet-50's "bottleneck" block (He et al. 2016, Fig. 5 right): a 1 × 1 convolution reduces the number of channels, the 3 × 3 convolution works on fewer channels, a second 1 × 1 convolution restores them. A 1 × 1 convolution is a weighted sum across channels at each position - During backprop, the skip connection also carries the error signal straight to earlier blocks - A block's input and output must have the same shape to be added; where a block changes the size or the number of channels, the skip path gets its own 1 × 1 convolution
- Carandini and Heeger called divisive normalization a canonical neural computation: it explains contrast saturation, cross-orientation suppression and surround suppression in V1, and appears in olfaction, audition, attention and multisensory integration - In biology the pool is other neurons responding to the same image; in BatchNorm it is the same channel across other images in the minibatch and across positions, which is easy to compute on a GPU but has no counterpart in a brain that sees one image at a time - VGG dropped AlexNet's local response normalization because it did not help accuracy; BatchNorm came back as an engineering fix for training speed, not as a model of cortex
- Computed here: one channel of ResNet-50's first convolution, before its BatchNorm, on 64 Imagenette validation photographs (a 10-class subset of ImageNet), at every one of the 112 × 112 positions. Batch mean −0.81, batch variance 10.18 - The standard deviation is the square root of the variance σ²_B. ε is a small constant (1e-5 in PyTorch) that avoids dividing by zero - γ and β are learned weights, one pair per channel; they let the network undo the normalization where that helps - Ioffe & Szegedy (2015) introduced BatchNorm; with it, networks trained in far fewer steps. Their reason was the drift above; later work (Santurkar et al. 2018) found BatchNorm helps mostly by making the loss surface smoother, so larger learning rates work. Either way, the practical consequence is the train/eval difference on the next slide - A neuroscientist's question: is this divisive normalization (Carandini & Heeger 2012)? Not quite. BatchNorm divides by the spread across the images of a batch and freezes those numbers at test time; cortex divides each neuron's response by its neighbors' responses to the same image
- The figure is a schematic of one BatchNorm layer - The running mean and variance are buffers: tensors saved with the model, updated by each forward pass in training mode, read in evaluation mode. For the channel on the previous slide, ResNet-50 stores a running mean of 0.09 and a running variance of 13.84 - model.train() and model.eval() switch BatchNorm (and dropout) between the two behaviors. Whether gradients are computed is set separately, by requires_grad and torch.no_grad - PyTorch's running statistics are exponential moving averages: each training batch moves them 10% of the way toward that batch's values (momentum 0.1; this "momentum" is a BatchNorm setting, unrelated to the training lecture's optimizer momentum)