Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7
Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7
Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7
Computed: the curve fit's 10 points, after Bishop (2006), Fig. 1.7
Computed: one monkey IT site (Bao et al. 2020), 1,224 object images, 512 late-layer units of ResNet-18; a random fifth held out
RidgeCV); too small a λ overfits (held-out 0.28 vs 0.69)Computed: the same curve and noise, after Bishop (2006), Fig. 1.6
Computed: the same curve and noise, after Bishop (2006), Fig. 1.6
Computed: the same curve and noise, after Bishop (2006), Fig. 1.6
Computed: the same curve and noise, after Bishop (2006), Fig. 1.6
Computed: the same curve and noise, after Bishop (2006), Fig. 1.6

transforms.Compose([
transforms.RandomResizedCrop(224, scale=(0.5, 1.0)), # Crop and resize
transforms.RandomHorizontalFlip(), # Flip horizontally at random
transforms.RandomRotation(15), # Rotate by up to 15 degrees
transforms.ColorJitter(0.3, 0.3, 0.3, 0.05), # Vary color and brightness
transforms.ToTensor(), # Convert to channel-first floats in [0, 1]
transforms.Normalize(IMAGENET_MEAN, IMAGENET_STD), # Standardize channels
])


Computed: the tiger, four 11 × 11 Gabor filters, stride 1, padding 5, ReLU




Optional: a few research-trail entries over the semester. Any paper mentioned in any lecture qualifies.
Submit on Canvas → Minute papers → Minute Paper 12 (access code read out in class).
Write three brief points in your own words:
Credit for a thoughtful attempt, not for being correct.
- Exam 1 is Thursday in class; tonight's recitation is exam preparation. The third assignment is released on Thursday
- Last lecture stopped after shortcut learning. These are the slides it did not reach: it showed when a fit fails to carry over to new data, and these say what to do about it - Regularization is the umbrella term. The next slides show three kinds: a penalty on the weights (also called weight decay or L2 regularization), dropout, and more examples, real or made by altering the ones you have. Early stopping, from the learning curves, is a fourth - Then the weight penalty applied to a model of a neuron, ridge regression, which the second assignment used and the next assignment uses again
- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect
- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect
- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect
- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect
- Conventions differ by a factor of 2. PyTorch's weight_decay and the demo use (λ/2)‖w‖², and scikit-learn's LogisticRegression uses C = 1/λ on the summed loss. When you choose λ on validation data, only its order of magnitude matters - The polynomial fit is a linear regression on the powers of x (x, x², …, x⁹), fit by least squares. With the penalty added, it is ridge regression, the method the next slides apply to a model of a neuron and the assignments use as RidgeCV - Test RMSE falls from 0.86 without the penalty to 0.34 with λ = 10⁻⁷ or 10⁻³, and rises to 0.61 with λ = 1 - The polynomial with 10 coefficients and 10 points, from last lecture. Without a penalty the coefficients reach about a million in size; with λ = 10⁻³, about 17 - Large coefficients are what let the curve swing between the points. Keeping them small keeps the curve smooth - w is the weight vector: in the polynomial, all 10 coefficients, the constant term included, and the penalty covers every one of them. For a network the same penalty runs over every weight. λ is the strength of the penalty, a hyperparameter (elsewhere in the course λ is an eigenvalue). Choose λ on validation data, never on the test set - The second assignment added this penalty to a readout (weight_decay); the next assignment chooses its strength on validation data - scikit-learn's LogisticRegression adds this penalty by default. A PyTorch loop does not, unless you pass weight_decay to the optimizer; PyTorch then applies it to every parameter it is given, biases included - Bishop (2006), Fig. 1.7, shows the same effect
- The weight penalty of the previous slides, applied to a model of a neuron instead of a curve - What is the same and what is new between the two assignments. Second assignment: one IT site from Bao et al. (2020), features from one layer of a network, nested cross-validation, scored by R². Next assignment: electrodes in V1, V4 and IT from the THINGS Ventral Stream Spiking Dataset (two monkeys, photographs), features from each of ResNet-50's 16 blocks, the same StandardScaler + RidgeCV with one λ per electrode, scored by the noise-corrected correlation (Pearson r divided by the square root of the electrode's reliability), and the question is which block predicts which area best - An encoding model predicts a neuron's response to each image from a network's features. With the penalty λ‖w‖² added to the squared error, the fit is called ridge regression. It is scored by R² on held-out images - The target is one IT site's response to each of the 1,224 object images of Bao et al. (2020); the features are the 512 units of the trained ResNet-18's late layer. A random fifth of the images is held out and never seen by the fit - The model is a weighted sum of the features plus a bias, fit by least squares: linear regression. Run with a small enough step on a continuous target, the delta rule minimizes the same squared error - λ is chosen by leave-one-out cross-validation within the 980 training images (scikit-learn's RidgeCV). The held-out fifth only scores the chosen fit. The held-out curve on the right is drawn to show what the penalty does; it was not used to choose λ - R² = 1 − (squared error of the prediction) / (squared error of predicting the mean response). 1 is perfect, 0 is no better than the mean, and it can be negative. The baselines lecture used the squared correlation, which is close for a good fit - Right panel: with almost no penalty, 512 weights fit the training images closely (R² 0.85) and predict held-out images poorly (0.28): overfitting. As λ grows, the training fit gets worse and the held-out fit better, up to a point (0.69) - The second assignment fit this model with nested cross-validation: λ is chosen inside each training fold, and the held-out R² comes from the outer folds - The same model, the same `RidgeCV`, twice. In the second assignment the target was one IT site and the features one network layer; in the next, the targets are recording sites in three areas and the features come from each of 16 stages of a network, so the fit is repeated 16 times per site - One λ per site: each site gets its own penalty, chosen on its own training images. RidgeCV does this with alpha_per_target=True - Nested cross-validation keeps the two decisions apart: λ is chosen on training folds; the score comes from images that were not used to choose anything
- Dropout (Srivastava et al. 2014): each hidden unit is switched off with probability p (often one half) at every training step, a different random set each time. A unit cannot count on any particular other unit, so the network cannot build fragile combinations that fit single training images. At test time all units are on, with their outputs scaled so their average matches training. PyTorch does the scaling during training instead (kept units × 1/(1−p)), so in evaluation mode nothing is rescaled. AlexNet used dropout in its fully connected layers; without it, Krizhevsky et al. (2012) wrote, the network "exhibits substantial overfitting" - L1 penalizes the sum of the absolute values of the weights instead of their squares. It drives many weights to exactly zero, so the model uses fewer inputs - The penalties differ in detail, but they share one effect: they restrict which weight settings training can reach, so the network has fewer ways to fit the noise in its training data - In PyTorch: nn.Dropout(p=0.5) as a layer; model.train() switches it on and model.eval() switches it off. Weight decay is the weight_decay argument of the optimizer
- Last lecture: the same **degree-9 polynomial** fitted to 10, 15, 30 and 100 points; test RMSE 0.86 → 0.31 - With 10 points it passes through every one; with 100 it cannot, and follows the curve instead - <mark>More examples always help the same model</mark>: overfitting is flexibility relative to the number of examples; $\lVert\mathbf{w}\rVert$ falls from a million to 8,700 with no penalty at all - The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had
- The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had
- The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had
- The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had
- The slide from last lecture: the model is kept and data is added. Error on new points 0.86 with N = 10, 0.34 with 15, 0.35 with 30, 0.31 with 100. Training error rises from 0.00 to 0.28, toward the noise level of 0.30: with many points the curve can no longer fit the noise - A network's flexibility is harder to count than 10 coefficients, but the same trade holds: more training images make the same network overfit less. More labeled data is the most reliable fix for overfitting, when it can be had
- Labeled data is expensive, so the next best thing to more data is to make more from what you have - Each epoch shows a different random version of each image, which makes memorizing the training images harder - Flips that would change the label (a 6 and a 9, a b and a d) must be left out. Speech recognizers train on recordings with stretches of time and bands of frequency masked out (SpecAugment, Park et al. 2019); medical-image networks on rotated and re-lit scans - Krizhevsky et al. (2012) trained AlexNet on random 224 × 224 crops of each 256 × 256 photo and their mirror images, which multiplies the training set by 2,048, plus color changes - Photograph: THINGSplus, public domain (CC0)
- torchvision.transforms.Compose applies the list in order to each image as it is loaded; the random transforms draw new parameters every time, so the training set is effectively unbounded. The held-out and test loaders use only Resize, CenterCrop, ToTensor and Normalize - Krizhevsky et al. (2012) used random 224 × 224 crops of each 256 × 256 photo and their mirror images, which multiplies the training set by 2,048, plus color changes. The next assignment's fine-tuning uses a pipeline like this one - Augmentations that change the label must be left out: a flip turns a b into a d and a 6 into a 9. Speech recognizers mask stretches of time and frequency (SpecAugment); medical-image networks rotate and re-light scans
- The networks of the last three lectures connected every unit to every input. For images that is wasteful, and it misses something: an object can appear anywhere. The next slides turn the dense layer into a convolution, the operation every vision network since has used
- Left: a two-layer perceptron over a 6 × 6 image: every pixel to every hidden unit (216 weights, all different), every hidden unit to every output. Right: a convolutional layer: each hidden unit reads a 3 × 3 patch, and the nine weights (color-coded) are the same for every unit, so the layer has nine weights instead of 216 and detects the same feature wherever it appears. A second convolutional layer reads patches of the map in the same way - Local connections and shared weights are the two changes that turn the perceptron into a convolutional network; the next slide counts what that saves, and the ones after show it as one operation, the convolution
- Counts are weights only, without biases. The image size is the 224 × 224 × 3 input stated in Krizhevsky et al. (2012) - A dense (fully connected) layer is the MLP layer of the lecture on learning the weights: every unit has one weight for every input value - Reusing a filter is called weight sharing (also parameter sharing): the 96 filters are the only first-layer weights, however many positions they are applied at - A dense unit could learn a local pattern, but it would have to learn it separately at every position, from many more examples
- Retinotopic organization: neighboring points of the visual field are handled by neighboring cells, so cortex holds a map of the image, with the same local machinery everywhere - On a computer the same answer comes from one filter applied at every position in turn, so the weights exist once: weight sharing. Counts are weights only, without biases, for the 224 × 224 × 3 input of Krizhevsky et al. (2012): 96 × 11 × 11 × 3 = 34,848, against 96 × 150,528 = 14,450,688 for dense units - Both the template and each window have their mean removed and are scaled to unit length, so the dot product is a cosine: 1.00 at the position the template was cut from, lower elsewhere. The template-matching idea is the model neuron's: among inputs of the same length, wᵀx is largest for the input that looks like w
- The window's patch has its mean subtracted before the dot product, so the filter reports contrast, not brightness; ReLU sets negative values to 0 in the map - The vertical filter responds along the tiger's flank; the horizontal one along the water's edge; the center–surround filter marks spots and edges of any orientation - The map has 90 × 90 positions, the image 96 × 96: no padding, stride 1, as the next slides define - The demo is on the course site with the other demos
- A **filter** (or **kernel**): a small grid of weights, a template, like a **simple cell**'s receptive field. At each position, multiply by the pixels under it and sum; a **ReLU** sets negatives to 0 - The result is a **feature map**: how well each part of the image matches the filter - Same weights everywhere: shift the image, and the map shifts with it (**equivariance**) - Computed here: the tiger (gray levels, 224 × 224) correlated with one 11 × 11 Gabor filter at stride 1 and zero padding 5 (both defined on the stride-and-padding slide), then a ReLU. The circle marks the largest value of the map, at the boxed image position. This oriented filter responds along the tiger's stripes that share its angle - Each value of the map is one unit's response; all the units of one map share the filter's weights - Equivariance: the map moves when the image moves. Invariance: the output does not change at all. Pooling adds invariance, a few slides on - PyTorch's Conv2d computes this sliding dot product without flipping the filter, which mathematicians call cross-correlation; deep-learning papers call it convolution all the same
- One filter slid over the image gives one feature map. A layer has many filters, and their maps stack into a volume: height × width × channels, one channel per filter. The next layer's filters then read all channels at once, which is why their weights have a depth of C_{l−1} in the notation that follows - Computed here: four Gabor filters at 0°, 45°, 90°, 135° on the tiger in gray levels; each map lights up where the stripes of the image run along that filter's orientation
- $\fmap{l}$: the layer's **feature maps**, $C_l$ of them, one **channel** per filter; $H_l \times W^{\mathrm{sp}}_l$ is each map's height and width - $\textcolor{#B5396B}{\kernel{l}}$: <span style="color:#B5396B"><b>the layer's filters</b></span>, each $k \times k \times C_{l-1}$, one slice per input channel · $\conv$ slides them over $\fmap{l-1}$ · $\textcolor{#3F7A6B}{\mathbf{b}^{[l]}}$: <span style="color:#3F7A6B"><b>one bias per channel</b></span> · $\textcolor{#7A4FA3}{g}$: <span style="color:#7A4FA3"><b>ReLU</b></span> - It is the MLP layer $\mathbf{a}^{[l]} = g(\mathbf{W}^{[l]}\mathbf{a}^{[l-1]} + \mathbf{b}^{[l]})$ with each unit wired only to a $k \times k$ patch of all $C_{l-1}$ input maps, and the same weights at every position. Weights per layer: $k \cdot k \cdot C_{l-1} \cdot C_l$ - The input image is the first stack, A^[0], with three channels (red, green, blue) - One filter spans every input channel: a first-layer filter on a color image is k × k × 3; a filter in a layer whose input has 64 channels is k × k × 64. Each filter makes one output channel - Per element: A^[l] at row i, column j, channel c is g of (the sum over the filter's rows u, columns v and input channels c′ of K at u, v, c′, c times A^[l−1] at i+u, j+v, c′, plus b_c) - W^sp is the spatial width; the superscript keeps it apart from the weight matrix W - In PyTorch, nn.Conv2d(C_in, C_out, kernel_size=k) stores the filters as a tensor of shape (C_out, C_in, k, k)
- From here on, the brain: the behavior to explain, the areas from the retina to IT, and Hubel and Wiesel's cells, which turn out to compute the two operations of the convolutional network, a weighted sum and a pooling - V1 sees edges in small patches; people recognize whole objects in a glance. The rest of today asks what happens between the two, in the brain and in a network - The brain half follows the ventral stream: center–surround cells, then V1's simple and complex cells, then the areas above V1. On the way it asks a question about learning: V1's oriented receptive fields can be learned from natural images alone, with no labels, by a sparse code - The network half prepares the next assignment: convolution, pooling, residual networks, BatchNorm, and reusing a pretrained network
- Core object recognition: naming the object in a single glance, across changes in position, size and background (DiCarlo, Zoccolan & Rust 2012) - The movie: ten photographs from the animal / non-animal set of Serre, Oliva & Poggio (2007), five animals and five distractors at the four viewing distances; each is shown for two frames at 30 frames per second, about 67 ms, after a fixation cross, then a two-second window to answer, then the answer - Thorpe, Fize & Marlot (1996): photographs flashed for 20 ms, a go/no-go response to animals; 94% correct. The EEG over frontal cortex differed between animal and non-animal trials from about 150 ms after the image - Too fast for eye movements or many back-and-forth loops: recognition rides mostly on one feedforward sweep through the visual system. Feedback connections exist everywhere; the speed limits how much they can contribute to the first answer
- Cauchoix, Crouzet, Fize & Serre (2016): two monkeys trained on the rapid animal / non-animal task with the same photographs people see; intracranial electrodes over intermediate and higher ventral areas (V4, posterior IT). A linear readout of the recorded activity reported the category early enough to drive the behavioral response, and the images the readout got wrong were the ones the monkeys got wrong - The movie: one monkey at the screen, photographs flashed one after another, responses at the box
- The figure is DiCarlo, Zoccolan & Rust (2012), Fig. 1: the macaque ventral stream (A) and its areas stacked with their typical latencies and approximate neuron counts (B). AIT, CIT, PIT: anterior, central and posterior IT - LGN: lateral geniculate nucleus, the relay between retina and V1. IT: inferior temporal cortex - Hung, Kreiman, Poggio & DiCarlo (2005): about 100 IT neurons, over windows as short as 12.5 ms, carried enough information for a linear classifier to report object identity and category, and the readout generalized over positions and sizes - The geometry lecture called this untangling (DiCarlo & Cox 2007): in pixels two objects' images cannot be separated by a flat (linear) boundary; in IT they can - The question for the rest of the lecture: what operation, repeated, gets from edges to objects?
- Kuffler (1953) recorded retinal ganglion cells of the cat with small spots of light: an on-center cell fires to light in a small central region and is suppressed by light in the ring around it; off-center cells do the opposite. LGN cells, one synapse on, have the same organization - Because center and surround cancel, a uniform field does nothing and an edge or a spot does: the retina already computes contrast. These are the inputs Hubel and Wiesel's cortical cells are built from, on the next slides
- The original Hubel and Wiesel film from the late 1950s and 1960s, shown in the first lecture: the experimenter moves a projected bar of light over a screen while one neuron in cat primary visual cortex is recorded, its spikes played as clicks - The cell is silent for most positions and orientations and fires briskly for one orientation at one place: its receptive field and its orientation preference, found by hand before computers ran the experiments - Torsten Wiesel describes the experiments at iBiology; Hubel and Wiesel shared the 1981 Nobel Prize in Physiology or Medicine for this work
- Hubel and Wiesel (1962) mapped receptive fields in cat visual cortex with spots and bars projected on a screen, marking by hand where light excited the cell (crosses) and where it inhibited it (triangles). Circular maps (a, b) are the center–surround cells of the LGN and layer 4; the elongated ones (c–g) are simple cells, selective for the orientation of a bar - The schematic on the right is the textbook summary of such maps: an ON stripe between OFF flanks (or the reverse), one orientation per cell. A Gabor function, a grating inside a Gaussian window, fits these maps with a few numbers (orientation, stripe width, phase, window size; Jones and Palmer 1987); the second assignment fit Gabors to network units the same way - The cell's response falls off smoothly as a bar is rotated away from its preferred orientation: its orientation tuning curve, typically 20–40° wide at half height
- What Hubel and Wiesel found, in one picture, before the mechanisms: center–surround inputs from the retina and LGN, simple cells built by summing rows of those inputs, complex cells built by pooling simple cells of one orientation at nearby positions - Two operations, then: a weighted sum (selectivity, the simple cell) and a pooling (tolerance, the complex cell). A convolutional layer is the weighted sum applied at every position; max-pooling models the complex cell. The Neocognitron and HMAX repeat the pair up the hierarchy
- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights
- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights
- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights
- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights
- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights
- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights
- Hubel and Wiesel's proposal, still the textbook model: several LGN cells whose centers lie in a row converge on one cortical cell; a bar of light along the row drives all of them at once, so the cortical cell fires strongly. Tilt the bar and it crosses fewer centers and more inhibitory surrounds, so the input drops; the cortical cell, which needs most of its inputs active, stops first - Computed on model cells: each LGN rate is the bar's overlap with a center-surround kernel, rectified; the simple cell fires when the sum of its four inputs exceeds a threshold. Frames at 0°, 15°, 30°, 60°, 90°; the simple cell's rate is 1.00, 0.14, 0, 0, 0 - The weighted-sum neuron of the first lectures models this cell: LGN responses are its inputs, connection strengths its weights
- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks
- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks
- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks
- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks
- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks
- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks
- Hubel and Wiesel's second wiring proposal: a complex cell pools several simple cells of the same orientation whose fields lie side by side, so it keeps the orientation selectivity and loses the position selectivity. A bar of the right orientation drives it wherever it falls within the pooled fields - Computed on model cells: five simple cells, fields 0.6 apart so neighbors overlap by about half a field; each responds as a Gaussian in the bar's position (width 0.25); the complex cell sums the five. Its response stays high across all five fields, dipping a little between them, where each simple cell covers only its own - Two steps: a weighted sum builds selectivity (the simple cell), pooling builds tolerance (the complex cell). The rest of today repeats this pair up the hierarchy and into networks
- Today: regularization, the move from the perceptron to the convolutional network, and the visual system up to V1: center–surround cells, Hubel and Wiesel's simple and complex cells, and how each is wired - Lecture 11 stopped after the simple-to-complex pooling frames. Next lecture starts with the rest: the ice-cube model, the areas above V1, the Neocognitron and HMAX, more on convolution (several filters, stride and padding, pooling), then the slides already held over (the mini-HMAX, ResNet-50, reuse, what trained networks learn, scale), then explainability
- Yamins, Hong, Cadieu, Solomon, Seibert & DiCarlo (2014), Performance-optimized hierarchical models predict neural responses in higher visual cortex - Zhuang, Yan, Nayebi, Schrimpf, Frank, DiCarlo & Yamins (2021), Unsupervised neural network models of the ventral visual stream - Konkle & Alvarez (2022), A self-supervised domain-general learning framework for human ventral stream representation - Kar, Kubilius, Schmidt, Issa & DiCarlo (2019), Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior - Jinsi, Henderson & Tarr (2023), Early experience with low-pass filtered images facilitates visual category learning in a neural network model - Lu, Thorat, Cichy & Kietzmann (2026), Adopting a human developmental visual diet yields robust and shape-based AI vision - Also cited today and worth reading: Kuffler (1953) and Hubel & Wiesel (1962) on receptive fields, Fukushima (1980) on the Neocognitron, Serre, Oliva & Poggio (2007) on rapid categorization - Before posting, skim the existing entries on your paper; your post must add something not already said
- Second half of the reading list; any paper mentioned in any lecture qualifies for the research trail
- Submitted on Canvas (Minute papers → Minute Paper 12); credit for a thoughtful attempt