


Computed · a petunia the network calls morning glory (p = 0.63)
Computed · 50 noisy copies · same network and photograph
Computed · same network and photograph · blurred copy: Gaussian blur, sd 10 pixels
Computed · baseline: a blurred copy · 32 steps
Computed · region: the brightest 6% of the SmoothGrad map, grown to 7%
We keep the weights fixed and change the image until one channel of layer , averaged over its positions, responds as strongly as possible:



TVSD (Papale et al. 2025): 16 of the 100 test photographs

Reproduced from Serre, Annu Rev Vis Sci (in submission), Fig. 4 (draft); Triple-N and TVSD scores computed for the review
Optional: a few entries over the semester; any paper from any lecture qualifies.
Submit on Canvas → Minute papers → Minute Paper 14 (access code read out in class).
The minute paper asks you to recall, in your own words, what you just heard. Retrieving it is what strengthens the memory; copying a slide does not. It is not a quiz.
A trend I have noticed: papers that discuss slides I never reached, and papers written while I was still lecturing. Both miss the point. From today, a paper about material not covered in class, or written before the class is over, loses credit.
Write three brief points:

Reproduced from Kamitani & Tong (2005), Fig. 2a
- Every method in the first half of today uses backpropagation from the training lectures with one change: the gradient is taken with respect to the image's pixel values, and the weights stay fixed. Adversarial images and feature visualization then step the pixels; attribution maps only read the gradient
- Training, from the backpropagation lectures: the gradient lists one slope per weight, and gradient descent takes a small step against it, again and again - The figure is the smallest network there is: one weight w times one input x, scored with the logistic loss for the correct class. Training (magenta) starts from a poor weight, w = 0.15, and takes eight gradient steps on w with x fixed, to w = 2.62 (loss 0.63 to 0.12). The attack (blue) starts from the trained network and takes five fixed-size steps on x along the sign of the gradient, up the loss, with w fixed (loss 0.12 to 1.98). The dashed line, wx = 0, is the decision boundary; below it the answer is wrong - With this loss there is no finite best weight: the loss keeps falling as w grows, so "trained" here means a low loss, and no finite weight is the best one - Up the gradient is the direction in which the loss rises fastest for a small step, so the attack maximizes the loss over the image, within a budget on how much each pixel may change, while training minimizes it over the weights. Adversarial training combines the two: minimize over the weights the loss of the worst image within the budget - Why that direction is also the shortest way to the boundary: for a linear readout the loss depends on x only through w·x, so the gradient with respect to x is along w, the normal to the boundary, and the boundary itself is a contour of the loss. For a deep network this holds only locally, step by step; the figure has a single input dimension, so the only choice is up or down - Weights and pixels enter the same chain of multiplications, additions and ReLUs, and the chain rule runs back to either. Training stops at the weights; the methods today run one step further, to the pixels - 25,557,032 weights in torchvision's ResNet-50; the input is a 224 × 224 color image
- Every row computes a gradient by backpropagation. The rows differ in what is differentiated, with respect to what, and what is done with the result. Training goes down the loss by changing the weights, starting from random ones. The other three never change a weight. Each asks the network a different question: what breaks it, which pixels it used, what a unit responds to - An adversarial example goes up the loss of the correct answer, within a budget small enough that people cannot see the change: next slide - An attribution map takes no step and changes nothing: the gradient at the photograph is itself the map. It says which pixels would change the class score most if they changed a little - Feature visualization does not start from a photograph: it starts from noise and climbs one channel's mean activation for many steps, so the result is a synthesized image. Neuroscientists run the same optimization on a model of one neuron and call the result its maximally exciting image (MEI)
- The network alternates between conv blocks, which hold the weights, and their outputs, the feature maps. Only two stacks are drawn and labeled, $A^{[l]}$ after a middle block and $A^{[L]}$ after the last, because later slides use them: Grad-CAM stops the gradient at the last layer's feature maps A^[L], and feature visualization starts it at an intermediate layer's A^[l]. Training uses neither; it needs only the weights - The objective is the loss ℓ(z, y); the gradient runs back and stops at every weight. This is backpropagation as in the training lectures, drawn for a convolutional network: one forward pass computes the loss, one backward pass gives a slope for every weight, and gradient descent changes the weights a little - On the slides that follow the weights are locked. The backward pass is the same; it runs one step further, to the pixels, or stops earlier, at a layer
- In the schematic the change is drawn as random signs. The next slide shows the real one: the sign of the gradient for a panda - x + ε · sign(∇ₓ ℓ) is the fast gradient sign method of Goodfellow et al. (2015). The sign comes from the budget: the change must stay invisible, so each pixel may move by at most ε. Under that rule the first-order best step moves every pixel by the full ε, up if its slope is positive and down if negative; a steep slope and a shallow one get the same ε. With a budget on the total length of the change instead, the step would follow the gradient itself, steep pixels moving more. The demo after it repeats the step ten times within the same budget
- Goodfellow, Shlens & Szegedy (2015), Explaining and harnessing adversarial examples. The fast gradient sign method: x + ε · sign(∇ₓ loss), with ε = 0.007 of the pixel range. Later today a gradient with respect to the pixels, of the logit instead of the loss, is the saliency map: there it explains the answer instead of changing it - The budget is a norm of the change δ = x' − x. L∞: the largest change to any one pixel, the panda's budget and the demo's; L2: the length of the whole change vector, the assignment's. The two budgets give different worst images, and a network robust under one need not be robust under the other - Robustness is read from the budget: the smallest ‖δ‖ that flips the answer, or the accuracy that survives attacks within a fixed ε. Random noise of the same size almost never flips the answer; the gradient finds the one direction that does - People are not fooled by perturbations this small; with a very brief presentation, larger perturbations made for many networks at once can bias human choices slightly (Elsayed et al. 2018)
- Madry et al. (2018), Towards deep learning models resistant to adversarial attacks; Salman et al. (2020), Do adversarially robust ImageNet models transfer better? The assignment's robust ResNet-50 is theirs - The inner max is solved approximately, by a few gradient steps up the loss on the pixels, each followed by a projection back into the budget; the outer min is a gradient step down the loss on the weights. Backpropagation used twice per step, once to the pixels and once to the weights - The attacks are regenerated at every step for the network as it is now, so they keep targeting its current weak spots - Robust networks' gradients look more like the object, and their representations predict some neural data better; the assignment's last part asks whether the robust ResNet-50 predicts IT better than the standard one
- Each setting is precomputed: ten gradient steps up the loss of the network's own answer on the original, each pixel kept within the attack size of its original value (projected gradient descent with an L-infinity budget). The probabilities are the network's on the attacked image - Standard ImageNet ResNet-50 with the flower readout: the petunia it calls morning glory (0.63) becomes bougainvillea (1.00) at half a gray level out of 255; the passion flower and the rose flip as fast - Robust ResNet-50 (Salman et al. 2020, adversarially trained) with its own flower readout, the one in the assignment: it calls the petunia petunia (0.91) and holds it up to 2 out of 255 (0.59); the passion flower holds to 6 out of 255 - Switch networks at the same attack size: the change for the robust network is not noise: amplified, it shows edges and petal shapes, because pushing a robust network's answer takes changes to what the object looks like. This is why robust networks' gradients make cleaner attribution maps and feature visualizations - The demo is on the course site with the other demos
- These slides ask which pixels drove one network's decision, the question of the assignment's third part - The gradient that built the adversarial example, read at the photograph instead of stepped along, explains a decision - Occlusion first, then the gradient at the pixels (saliency), why that map is noisy and why it misses detail, Grad-CAM, and two tests of any map
- Reproduced from Zeiler & Fergus (2014), Fig. 7, columns (a) and (d), laid out in a row, color bars dropped (red is high, blue low): in each pair, left, the input with the gray square at one position; right, the probability of the correct class (Pomeranian, car wheel, Afghan hound) with the square centered at each position - Covering the Pomeranian's face drops the probability of "Pomeranian" sharply; covering other parts of the image barely changes it. For the third image, covering the dog matters and covering the man's face does not, although the man's face drove the strongest feature map in the network's top layer - An occlusion map asks what a decision depends on, by changing the input. A unit's strongest feature map asks what a unit responds to. The two need not agree - Why the caveat matters: the network never saw gray squares in training, so the square can lower the score on its own - Two checks. Change the fill: a blurred copy, noise, or the region filled in from its surroundings (inpainting, a few slides on). If the map's bright region stays put across fills and square sizes, the drop comes from what is hidden. And place the same square on a region the map calls unimportant: if the score drops there too, the square itself is doing the work - Gradient-based attribution, after the demo, asks the same question without rerunning the network for every position
- Sweep: the square visits every position and draws the map for the selected class. Unlike the previous slide, the map here is the drop in probability (clean minus covered), so bright marks the regions that matter - Clean image: English springer 0.625, hen 0.018, Irish water spaniel 0.130, Sussex spaniel 0.065, miniature poodle 0.030 (ImageNet ResNet-50; 1,795 reruns precomputed: 16, 32 and 64 px squares at every 8-pixel position) - A 32 px square on the dog's face (row 40, column 136) sends springer 0.625 → 0.000 while hen rises to 0.371 and miniature poodle to 0.388; a 64 px square there gives hen 0.835 - A 32 px square on the hen's head (row 128, column 40) barely moves springer (0.587): the decision does not depend on the hen - The hen map's own peak is only 0.018, so on the fixed 0–0.625 scale it is flat: read a map together with the score it explains - The demo is on the course site with the other demos
- In the lecture on learning the weights, backpropagation computed the gradient of the loss with respect to the weights. Here the same chain rule gives the gradient of one logit with respect to the pixels; the weights stay fixed - For a linear neuron this map would be its receptive field, the weights; for the network it is the class unit's receptive field measured at this one photograph, and it changes from photograph to photograph, unlike a map from reverse correlation, which averages over many stimuli - The logit is used instead of the probability, so the map does not also reflect the scores of the other classes through the softmax - Absolute value: a pixel matters whether brightening or darkening it raises the score - In practice: one forward pass, one backward pass, then the largest of the three color channels at each pixel. In PyTorch, `x.requires_grad_(True)`, then `logits[0, c].backward()`, and the map is `x.grad.abs()`
- Training's backpropagation with two changes: the logit z_c at the top instead of the loss, and the pixels at the bottom instead of the weights. The map is large where a small change of that pixel moves z_c most - Why z_c and not the loss: the loss also depends on the label and on every other class through the softmax; the logit z_c asks about one class only - Which class c: by default, the predicted class, because the map should explain what the network did; it needs no label, so it works on any image. The assignment does this. For a mistake, also compute the map for the correct class and compare the two: the predicted-class map shows the evidence that drove the wrong answer, the correct-class map shows whether the network found evidence for the right one, and where. The petunia here is such a mistake: the map is for morning glory, the network's answer
- In the backpropagation lecture the error of a unit was δ = ∂ℓ/∂z, the slope of the loss with respect to its weighted sum, and the hidden-unit formula computed it from the errors of the units above: multiply each by the weight on the connection and by the slope of the activation, then add. Here the quantity at the top is the logit z_c instead of the loss, so the output layer's error is simply 1 for unit c and 0 for the others; everything below is the same recipe - Training uses the first hidden layer's errors once more to get the weight gradient, error times input, and stops. Saliency uses the same errors to get the pixel gradient: pixel k enters the weighted sum of every unit j of the first hidden layer, the first convolution, with weight w_jk, so its slope is the sum of δ_j w_jk over those units. A pixel has no activation function of its own, so no g′ term appears - In a convolutional layer the sum over j runs over the units whose receptive field contains the pixel, and w_jk is the filter weight at that offset: the backward pass through a convolution is a convolution with the flipped filter - Autograd does this bookkeeping: `x.requires_grad_(True)`, a forward pass, `logits[0, c].backward()`, and `x.grad` holds the map. Training reads the weights' `.grad` instead; one call to `.backward()` fills both
- Computed for the course: the flower network and the petunia of the previous slides, for the class it predicts. One fixed Gaussian noise pattern (sd 0.1 in [0, 1] units, the noise SmoothGrad uses) is scaled down and added to the photograph; the horizontal axis gives the typical change of each pixel's value, out of 255 - A saliency map is the gradient at one image. ResNet-50 in evaluation mode is built from ReLUs, max-pooling, averages and fixed rescalings, so its logit is a piecewise-linear function of the pixel values: flat pieces glued together, with a different slope on each piece. Nearby images sit on different pieces, so their gradients differ, while the logit itself changes smoothly - So a single saliency map mixes the pattern that matters with jumps that would change at the next image. The fix on the next slide averages the maps of many nearby images
- The noise here is much larger than on the previous slide: sd 0.1 in [0, 1] units, about 25 gray levels, as in the assignment. Each copy lands on a different linear piece of the network, so averaging over copies averages over the jumps. It is not a blur of one map: each copy's map is computed by its own backward pass - SmoothGrad: Smilkov et al. (2017); the assignment uses 50 noisy copies, noise 0.1 - The maps on these slides are raw; display tools often smooth them
- Each point of the curve is the image (1 − a) × blurred copy + a × photograph, for a from 0 to 1: every pixel's value moves in a straight line from its blurred value to its value in the photograph - The gradient at the photograph answers only: if the pixel values change a little from here, how much does the logit change? On the flat part, almost not at all. It cannot say that the detail, compared with no detail, raised the logit - Saliency and SmoothGrad both read slopes near the photograph, so both share this limit. The next method uses the whole path
- x − x₀ is fixed along the path, but it is not one number: it is a whole image, one value per pixel, the difference between the photograph and the baseline. It does two things. It turns a slope into a change of the logit, as slope × distance does on a hill; adding these over all pixels gives the whole rise from the baseline to the photograph. And a pixel that is the same in the photograph and the baseline gets no credit, however steep its slope - The path is the one of the previous slide. Each pixel's credit is its average slope along the path times the change of its value (⊙: pixel by pixel). Summed over all pixels, the credit equals the logit at the photograph minus the logit at the baseline, 10.69 − 2.83 = 7.86, up to the error of using m = 32 steps - The baseline says what "no evidence" means. A black image is the original paper's and Xplique's default, used in the assignment's bonus; a blurred copy keeps the colors and removes the detail, so the map credits the detail - Integrated gradients: Sundararajan, Taly & Yan (2017), ICML
- The gradient stops at the last feature maps; each channel's weight α_k is the slope of z_c with respect to channel k, averaged over positions; the map is the weighted sum of channels with negatives set to zero, 7 × 7 for ResNet-50, then enlarged to the image. Each channel of the last convolutional layer is a map of where one feature is present; the gradient says how much z_c rises when that feature is stronger. Weight each map by that sensitivity, add them up, and keep the positive part: the regions whose features push z_c up. Two indices: c is the class being explained, k runs over the channels of the last feature maps - The α values in the figure are illustrative, not computed. The assignment's bonus computes Grad-CAM for the flower network - Selvaraju et al. (2017), Grad-CAM: visual explanations from deep networks via gradient-based localization, ICCV
- Computed for the course: the flower network, the petunia, the probability of morning glory. The map's region is the brightest 6% of the SmoothGrad map, its largest connected patch, grown to 7% of the image. The square has the same area and is centered on the same patch, so it covers mostly the same pixels - All four removals take the dark center, and all four drop the probability, from 0.63 to between 0.15 and 0.00. Here inpainting drops it further than gray: the gray patch keeps a dark, roughly central blob, which the network may still partly read as the flower's center - Inpainting: OpenCV's Telea method. It fills the region from its border inward, each new pixel a weighted average of the known pixels near it - A drop alone does not test the map. The next slide adds the control
- Region: the brightest 6% of the SmoothGrad map's pixels, its largest connected patch, grown slightly: 7% of the image - Removing it drops p(morning glory) from 0.633 to 0.001; the 30 same-shape controls leave it between 0.50 and 0.82, median 0.67, none at or below the map region's value. Here the map and the intervention agree - Why a control: any edit makes the image less natural and can lower the score by itself. The control removes the same amount of image, of the same shape and size, placed where it misses the map's region - Same logic as a lesion or a muscimol inactivation with a sham control: remove a part, compare with removing an equal part elsewhere. As with a lesion, the edit can add something new (here a smooth pink patch), which is what the control is for - A smoother map is not a more accurate one: SmoothGrad looks cleaner than saliency, but only a test like this one says whether its region is the one the network uses - The assignment uses a simpler test: the share of the map's top pixels on the flower's segmentation mask, against a map that ignores the network
- Computed for the course: the same network and photograph. The previous test removed one region. Deletion tests the whole ranking: pixels go in the order the map gives them, 10% at a time, each replaced by the blurred copy of the photograph, and the class probability is recorded after every step. Any map can be scored this way, so the maps can be compared on one photograph - The area under the curve summarizes it: a map that finds the pixels the network uses drops the probability within the first few percent and gets a small area; a random map removes pixels in no useful order and gets a large one - Blurring 10% of the pixels can also raise the probability (the random and integrated-gradients curves rise at the first step): the blurred copy removes detail the network read against morning glory. Only the fall over the whole curve counts - Insertion is the mirror test: start from the blurred copy and put pixels back in the map's order; a good map restores the score fast - The rankings depend on the photograph: do not assume this order holds elsewhere. Occlusion's map is itself made by removing regions, which is what deletion scores, so it has a head start
- The same tool, run on a model of an IT neuron, makes the images of the assignment's last part
- The layers above layer l play no part. They are grayed out because the objective does not depend on them: the backward pass starts at layer l - The result panel shows a real visualization: channel 959 of ResNet-50's layer3, made with MaCo like the images that follow
- Maximally exciting image, MEI: the name neuroscientists use for the image that drives a neuron, or a model of one, hardest. The assignment's last part computes MEIs for models of IT sites. Bashivan et al. (2019) showed such images to a monkey and drove V4 sites past every natural image; it is on the reading list - Tanaka's reduction method searched by hand while recording; here the gradient does the search. A neuron gives no gradient, so neuroscientists fit a model of the neuron and optimize the model (Walker et al. 2019, Bashivan et al. 2019), or search without gradients, by evolving images from the neuron's firing (Ponce et al. 2019) - The fourth row of the uses table: the gradient to the pixels, many steps, to maximize one channel - Typical penalties: small random shifts of the image at every step (jitter), blurring, and penalizing fine high-frequency detail - The images on the next slide use MaCo (Fel et al. 2023), which keeps the image's spectrum at that of natural images and optimizes only the phase - Unit: one channel of a layer, its activation averaged over positions
- Computed for the course with MaCo (Fel et al. 2023), four units per layer; first layer: the filters themselves, not optimized images - AlexNet (Krizhevsky et al. 2012), torchvision ImageNet weights. Units: the four highest-scoring of 16 units chosen for their variance over 500 ImageNet photographs, not for any category, so no unit was picked because it looks like a face or a car - First layer: the 11 × 11 filters themselves (weights), shown with zero as mid-gray. Middle and last: MaCo images, each the input that drives one channel's mean activation hardest - A hierarchy like the one from V1 to IT: small oriented features first, then textures, then larger parts.
- Computed for the course with MaCo (Fel et al. 2023), four units per layer; first layer: the filters themselves, not optimized images - ResNet-50 (He et al. 2016), torchvision ImageNet weights, the encoder of the flower network and of the assignment. Units chosen as for AlexNet; the last-block units are 179, 1975, 1352 and 1839; unit 179 comes back with its top photographs after the two transformer slides - First layer: the 7 × 7 filters. At this size they are blurry, but color-opponent centers and an oriented edge are visible - Face units: none of these sampled units is a face unit. ImageNet has few face categories; studies that find face-selective units search for them, for example by the images that drive each unit
- Computed for the course with MaCo (Fel et al. 2023), four units per layer; first layer: the filters themselves, not optimized images - ViT-B/16 (Dosovitskiy et al. 2021), supervised on ImageNet. A vision transformer's first layer cuts the image into non-overlapping 16 × 16 patches, a grid of 14 × 14 for a 224-pixel image, and turns each patch into a token: one linear projection maps the patch's 16 × 16 × 3 = 768 pixel values to 768 numbers. Attention then mixes tokens across the whole image from the first block on - Why no orientation: in a convolutional network, each first-layer filter is a small window slid over every position, so a unit has to be a local detector, and training on natural images makes it an oriented edge detector, as V1 cells are. A patch projection is not slid over overlapping positions and has no built-in preference for neighboring pixels: each output channel mixes all 256 pixels of a patch. Orientation is spread over many channels instead of being the preferred direction of one; the principal components of the projections do look like smooth, Gabor-like basis functions (Dosovitskiy et al. 2021, Fig. 7) - Position is not built in either: a learned position embedding is added to each token - The last block's images are less part-like than ResNet-50's: each token can draw on the whole image, so a unit need not respond to one localized part
- Computed for the course with MaCo (Fel et al. 2023), four units per layer; first layer: the filters themselves, not optimized images - DINOv2 ViT-B/14 (Oquab et al. 2024): a vision transformer with 14 × 14 patches, trained without labels to give matching representations to different crops of the same image - Its first layer has the same patch structure as ViT-B/16 and the same lack of oriented filters (previous slide) - Labels are not what makes textures and parts appear: a network trained only to recognize that two crops come from one image learns them too. The assignment's other self-supervised network, DINO ResNet-50, is a convolutional network trained the same way
- Three units of layer4 of the ImageNet ResNet-50. Unit 309: conical shapes (tents, volcanoes, yurts); unit 179's photographs (a bell, screws, a shark) share no single label: a unit can respond to several unrelated things - Top-activating images are the classic method of Zeiler & Fergus (2014) and Zhou et al. (2015): units for faces, text, wheels and fur appear in networks never told to find them - Top images: run the whole dataset through the network and keep the images that drive the unit hardest - ResNet-50 (torchvision weights), layer4, six top images from 50,000 ImageNet validation images
- How the concepts are found (CRAFT, Fel et al. 2023): take the last block's activations on crops of the class's photographs and factorize them into a few non-negative parts; each part is a direction in the block's activity, a concept, ranked by its contribution to the class score. Each concept is then visualized with MaCo and shown with the crops that activate it most - A shortcut (Geirhos et al. 2020): a feature that predicts the label in the training photographs without being the object. The network can be right for the wrong reason, and an accuracy score does not show it. A test: a tench photographed without hands, or a hand without a fish - Concept 6 is fish heads, the object itself. The site's importance percentages are each concept's share of the class score, not a causal test - Geirhos, Jacobsen, Michaelis, Zemel, Brendel, Bethge & Wichmann (2020), Shortcut learning in deep neural networks, Nature Machine Intelligence
- Quiroga, Reddy, Kreiman, Koch & Fried (2005), Invariant visual representation by single neurons in the human brain: recordings in epilepsy patients; one neuron fired for Halle Berry across photographs, a drawing and the letter string, another for Jennifer Aniston - Goh, Cammarata, Voss, Carter, Petrov, Schubert, Radford & Olah (2021), Multimodal neurons in artificial neural networks: in CLIP (Radford et al. 2021), a network trained to match photographs with their captions, many units in the last layers respond to a concept across photographs, drawings and text: people, places, emotions, holidays. The tiles are the article's own examples - Typographic attack: a Granny Smith apple photographed with a handwritten note "iPod" is classified as an iPod with high confidence; a photograph with the word "pizza" written on it is called a pizza. The units that read the concept also read its name
- Templeton, Conerly, Marcus, Lindsey, Bricken, Chen, Pearce, Citro, Ameisen, Jones, Cunningham, Turner, McDougall, MacDiarmid, Freeman, Sumers, Rees, Batson, Jermyn, Carter, Henighan & Olah (2024), Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet - A feature here is not one unit. Single units of a language model respond to many unrelated things, so the authors fit a dictionary of directions (a sparse autoencoder) to the activity of one layer and read each direction's top-activating inputs, the language analog of the top photographs of ResNet-50's units. The figure shows the inputs that activate the Golden Gate Bridge feature most strongly: photographs of the bridge and text naming it, in several phrasings - Driving the feature: add its direction to the activity, at several times its usual strength, and the model describes itself as the bridge and brings the bridge into unrelated answers. Served to the public for one day in May 2024 as Golden Gate Claude - No gradient here: the feature's activity is read directly and set directly. The questions are those of feature visualization and of the maximally exciting image: what drives it, and what it does to the output - The networks' layers form a hierarchy like the one from V1 to IT. Whether the blocks also match the areas is the question of the next section
- So far today: which pixels one decision used, and the image a unit likes most. Now the brain: score a network's blocks against V1, V4 and IT recordings with the readout and noise ceiling you already know
- Every tool on this slide came earlier in the course: the encoding model and ridge regression in the vision lecture, the noise ceiling and split-half reliability in the baselines lecture, RSA in the MDS lecture. Nothing new here, only the data - A recording site is one electrode, picking up the summed spiking of the few neurons near its tip (multi-unit activity) - Score used on the next slides: the Pearson correlation between a site's predicted and recorded responses on the 100 test photographs, divided by the square root of the site's split-half reliability (Spearman–Brown corrected): the noise-corrected correlation. The square root, because a reliability is already a fraction of variance - The THINGS Ventral Stream Spiking Dataset (Papale et al. 2025): monkeys F and N looked at 22,248 photographs of 1,854 object concepts from THINGS (Hebart et al. 2019), each seen once, and 100 test photographs seen 30 times, which give each site its reliability. Each site's readout is fit on 5,000 of the single-presentation photographs - Each site's readout: ridge regression from a block's features (each feature map average-pooled to 2 × 2, then flattened), λ chosen by cross-validation on the training photographs. A readout from layer4 has up to 8,192 weights (2,048 maps × 2 × 2), fitted on 5,000 photographs, so the ridge penalty is what keeps it from memorizing noise
- The assignment's last part does this for the last block of the standard and the robust ResNet-50 - Computed for this lecture on the TVSD test photographs, once for every block; raw medians, not divided by each area's peak. The y axis starts at 0.35 - Each site: a ridge model from the block's features (each map average-pooled to 2 × 2, then flattened) to the response, λ chosen by cross-validation on the training photographs; scored by the Pearson correlation on the 100 test photographs, divided by the square root of the site's split-half reliability (Spearman–Brown corrected). Squared, it is the fraction of the site's repeatable variance that the readout predicts - In monkey N the V4 and IT peaks are one block apart and their intervals overlap: read the shapes, not the peak block - Bands: 95% intervals over 1,000 bootstrap draws of the test photographs. Peaks with intervals: V4 F 0.680 [0.611, 0.729], N 0.706 [0.645, 0.750]; IT F 0.759 [0.721, 0.790], N 0.812 [0.735, 0.852]; V1 F 0.638 at layer1.0, N 0.662 at layer3.2 - V1 is predicted almost equally by every block up to the last group, so its curve does not say where V1 sits: each map is pooled to 2 × 2, while a V1 site sees a small patch. Fits that keep each map's positions place V1 at the early blocks (Cadena et al. 2019) - Training matters: on the first lecture's IT sites, a ResNet-50 with random weights reaches about 0.30 against 0.52 trained (baselines lecture) - The last blocks fall because they serve the 1,000 ImageNet labels, the reason the assignment fine-tunes layer4 instead of reading it out as it is
- Your question from the first minute paper: is AI getting closer to the brain, or further away? Two figures, then a discussion
- Neural alignment: a linear fit from a network's features to monkey IT responses (recordings from Arcaro et al. 2020), scored on held-out images, the same kind of score as the block-by-area curve. ImageNet multi-label accuracy credits every valid label of an image; the gray band is the range of human accuracy - Red: CNNs trained on ImageNet-1K; yellow: adversarially trained CNNs; green: convolutional transformers; blue: vision transformers, with lighter shades for internet-scale training and gray for vision-and-language models. Purple stars are models added in 2026 by Akash Nagaraj - Once accuracy passes about 0.9, more accurate models predict IT less well, and the trend holds within families. Scale alone is not bringing networks closer to the brain - Yellow, the adversarially trained CNNs, are robust networks like the one the assignment compares with the standard ResNet-50 as models of IT
- A draft figure from my Annual Review of Vision Science chapter: the cohort of Linsley, Feng and Serre (2026), 363 released ImageNet classifiers (supervised CNNs and transformers, self-supervised, image–text and adversarially trained variants), scored with the Brain-Score protocols against four targets - Targets: (a) spatial response maps of face-selective IT neurons (Arcaro et al. 2020); (b) Triple-N (Li et al. 2026), Neuropixels recordings through IT including the category-selective patches, scene photographs; (c) TVSD (Papale et al. 2025), the assignment's data; (d) human ClickMe feature maps (Linsley et al. 2019). ρ is the rank correlation between accuracy and score, over all 363 and over the 84 above 92.7% accuracy - Scores are not divided by their ceilings here (the dashed line is each dataset's noise ceiling, or human-to-human agreement in d). Architecture and training vary together across the cohort, so this says whether accuracy gains bring alignment gains; it does not isolate what causes them
- The lab is running these experiments on TVSD and Triple-N now; the chapter's "future issues" are the same questions at full scale: how the structure of visual experience shapes what is learned, how resource constraints shape computation, which algorithms networks acquire, how to tell apart models that predict the same responses, and whether such relations hold across tasks and species - Controversial stimuli (Golan, Raju & Kriegeskorte 2020): images synthesized so that two models disagree about the category; a person's or a neuron's answer then favors one model. Prince et al. (2026) found that models with similar predictivity differ in how well their images control neurons - Scope for a team: one dataset, one manipulation, one measurement, controls named before the run; the write-up says what would have falsified the hypothesis - The project brief on Canvas lists the datasets, the compute, and the deadlines (teams by 11/19, proposal 11/24)
- The lecture can stop here; the slides after the minute paper are optional reading - Next lecture: learning without labels, the last lecture of Theme 2
- Schrimpf, Kubilius, Lee, Ratan Murty, Ajemian & DiCarlo (2020), Integrative benchmarking to advance neurally mechanistic models of human intelligence - Bashivan, Kar & DiCarlo (2019), Neural population control via deep image synthesis - Conwell, Prince, Kay, Alvarez & Konkle (2024), A large-scale examination of inductive biases shaping high-level visual representation in brains and machines - Goh, Cammarata, Voss, Carter, Petrov, Schubert, Radford & Olah (2021), Multimodal neurons in artificial neural networks - Geirhos, Jacobsen, Michaelis, Zemel, Brendel, Bethge & Wichmann (2020), Shortcut learning in deep neural networks - Templeton et al. (2024), Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet - Muttenthaler, Greff, Born, Spitzer, Kornblith, Mozer, Müller, Unterthiner & Lampinen (2025), Aligning machine and human visual representations across abstraction levels - Fel, Rodriguez, Linsley & Serre (2022), Harmonizing the object recognition strategies of deep neural networks with humans - Papale, Wang, Self et al. (2025), An extensive dataset of spiking activity to reveal the syntax of the ventral stream - Also in the optional slides at the end, and worth reading: Kamitani & Tong (2005) on decoding orientation, Horikawa et al. (2013) on decoding dreams - Before posting, skim the existing entries on your paper; your post must add something not already said
- Submitted on Canvas (Minute papers → Minute Paper 14); credit for a thoughtful attempt - Write the paper after the lecture, about what was covered in class; a paper about slides not reached, or written during the lecture, loses credit
- MVPA: multivoxel pattern analysis, the fMRI name for decoding - Linear predictivity ran from network to neuron. Readouts also run the other way: what does an area's activity say about the stimulus? The geometry lecture read animacy from IT neurons this way - Each polar plot: the grating shown at its top left; the outline, over 22 test samples per orientation, how often the readout chose each of the 8 orientations - A voxel is one fMRI measurement, a small cube of cortex a few millimeters across - Orientation columns: V1 neurons preferring one orientation sit in patches about half a millimeter wide, smaller than a voxel - Each voxel averages many columns with a slight imbalance; the readout combines these small biases. That was the original explanation; later work found much of the signal in coarse maps across V1, such as a preference for orientations pointing at the fixation point (Freeman et al. 2011), and the question is still debated - The figure's panels are from participant S2, 400 voxels of V1 and V2 - Trained and tested on separate runs, as in the baselines lecture - Decoding accuracy says the information is present and linearly readable, not that the brain uses it - Haxby et al. (2001) founded the method: patterns across ventral temporal cortex distinguish object categories, even outside the regions that respond most to each category
- Horikawa, Tamaki, Miyawaki & Kamitani (2013), Neural decoding of visual imagery during sleep - Participants were woken repeatedly in early sleep and asked what they had seen; decoders were trained on their awake responses to photographs - Suggests dream content uses the same visual representations as seeing