Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 13 · Theme 2: Learning representations: from neurons to deep networks

Comparing networks with the brain

Thursday, October 29 · Fall 2026

Reader-friendly version — lecture content with figure descriptions

Backpropagation trains a network, and it can also question one

Hold the weights fixed and take the gradient with respect to the image

A network is a function of its weights and of its input

Contour map of the logistic loss of a one-weight network over the weight w (horizontal) and the input x (vertical). A magenta training path moves right along w at fixed x, from w = 0.15 to w = 2.62, loss 0.63 to 0.12. From the trained point a blue attack path moves down along x at fixed w, crosses the dashed decision boundary at x = 0, and the loss rises to 1.98

Computed · the smallest network: one weight, one input, logistic loss

  • A network computes y^=f(x; θ)\hat{\boldsymbol{y}} = f(\textcolor{#3E6D8E}{\boldsymbol{x}};\,\textcolor{#B5396B}{\boldsymbol{\theta}}). Its loss ℓ\ell depends on both. The network in the figure has a single weight, w\textcolor{#B5396B}{w}
  • In training, we keep the image fixed and change the weights to lower the loss, θ←θ−η ∇θ ℓ\textcolor{#B5396B}{\boldsymbol{\theta}} \leftarrow \textcolor{#B5396B}{\boldsymbol{\theta}} - \eta\,\nabla_{\textcolor{#B5396B}{\boldsymbol{\theta}}}\,\ell
  • In an attack, we keep the weights fixed and change the pixels to raise the loss. Backpropagation down to the pixels gives one slope per pixel, ∇x ℓ\nabla_{\textcolor{#3E6D8E}{\boldsymbol{x}}}\,\ell. A few small steps push the image across the decision boundary
  • ResNet-50: 25.6 million weights, 150,528 pixel values (224 × 224 × 3)

Backpropagation, four uses

use gradient of variable then steps
training loss ℓ\ell weights θ\textcolor{#B5396B}{\boldsymbol{\theta}} step down many
adversarial example loss ℓ\ell pixels x\textcolor{#3E6D8E}{\boldsymbol{x}} step up one, tiny
attribution map logit zcz_c pixels x\textcolor{#3E6D8E}{\boldsymbol{x}} the gradient is the map none
feature visualization one channel's activation pixels x\textcolor{#3E6D8E}{\boldsymbol{x}} step up many, from noise
  • Step: one gradient update. Logit zcz_c: class cc's score before the softmax. In the last three rows, the weights never change

Training: the gradient flows back to the weights

A schematic network, left to right: the petunia photograph, image x; conv blocks with weights theta; feature maps A of layer l; more conv; the last feature maps A of layer L, channels k; class scores z; and at the right, the objective box. Gray arrows carry the forward pass; a red arrow carries the gradient backward. Here the objective is the loss of z and the label y; the red arrow branches up into every conv block; the image is fixed and the weights change, theta gets theta minus eta times the gradient of the loss
  • Each conv block outputs a stack of feature maps. Only two are drawn, A[l]A^{[l]} and the last, A[L]A^{[L]}. In the next slides the network stays the same. We change what sits at the top, and where the backward pass stops

An adversarial example steps the pixels up the loss

A schematic network, left to right: the petunia photograph, image x; conv blocks with weights theta; feature maps A of layer l; more conv; the last feature maps A of layer L, channels k; class scores z; and at the right, the objective box. Gray arrows carry the forward pass; a red arrow carries the gradient backward. Here the objective is the loss, to go up; the weights are fixed and the red arrow runs all the way back to the pixels; below, the petunia unchanged to the eye, and x gets x plus epsilon times the sign of the gradient of the loss with respect to x
  • We take one small step up the loss. No pixel changes enough to see, yet the answer changes
  • Why only the sign? Each pixel may change by at most ε\varepsilon. The loss rises most if every pixel uses the full ε\varepsilon

Adversarial examples

Goodfellow et al. 2015, Figure 1: a photograph of a panda, classified panda with 57.7 percent confidence, plus 0.007 times a colorful noise pattern, the sign of the gradient of the loss with respect to the pixels, equals an image that looks identical to the panda but is classified gibbon with 99.3 percent confidence
  • After one small step along the gradient with respect to the pixels, the network calls the panda a gibbon
  • A norm measures the size of the change. Here it is the largest change of any pixel. In the assignment it is the length of the whole change. The smaller the change that flips the answer, the less robust the network

Adversarial training: learn from the worst case

min⁡θ    average over images⏟training    max⁡∥δ∥≤ε  ℓ(f(x+δ; θ), y)⏟attack\min_{\textcolor{#B5396B}{\boldsymbol{\theta}}}\;\;\underbrace{\text{average over images}}_{\text{training}}\;\;\underbrace{\max_{\|\boldsymbol{\delta}\|\le\varepsilon}\;\ell\big(f(\textcolor{#3E6D8E}{\boldsymbol{x}}+\boldsymbol{\delta};\,\textcolor{#B5396B}{\boldsymbol{\theta}}),\,y\big)}_{\text{attack}}

  • The inner max is an attack. It finds the change, within the budget, that raises the loss the most, the worst case. The outer min is ordinary training
  • At every training step, we attack the current network and train it on the attacked images. Training on random noise of the same size, or on attacks computed once at the start, gives much less robustness
  • It costs a few points of ImageNet accuracy. The assignment's robust ResNet-50 was trained this way, with an L2L_2 budget of ε=3\varepsilon = 3

Try it: attack the flower network

  • Try the standard network at 0.5/255, then the robust one at 2/255. Half a gray level fools the standard network. The robust one still gets it right

Attribution maps show which pixels a decision used

Occlusion, saliency and two fixes for it, Grad-CAM, and how to test a map

Occlusion: cover a region and watch the score

Three pairs side by side, each a test photograph with a gray square over part of it and, to its right, a map of the probability of the correct class as the square moves: a Pomeranian dog, low only where the square covers the dog's face; a racing car's wheel, low over the wheel; a couple with an Afghan hound, low over the dog and not over the people's faces

  • Occlusion needs no gradient. We cover part of the image with a gray square, run the network again, and repeat at every position. Each point of the occlusion map is the probability of the correct class with the square centered there. Dark regions are the ones the answer depends on
  • A gray square is an unnatural input, so the square itself can lower the score. Try other fills (blurred, noise, the region filled in from its surroundings) and other sizes. A region that really matters should matter for all of them

Try it: cover a region, watch five scores

  • Drag the square onto the dog, then onto the hen, and watch the two scores swap. Unlike the published map, this map shows the drop in probability, so bright means the region matters

Saliency: the gradient at the pixels

The petunia photograph, left, and its saliency map for the class morning glory, right: a speckle of bright pixels spread over the whole flower, densest at the dark center

Computed · a petunia the network calls morning glory (p = 0.63)

s  =  ∣ ∂zc∂x ∣\textcolor{#B23A2E}{\mathbf{s}} \;=\; \left|\,\frac{\partial \textcolor{#3F7A6B}{z_c}}{\partial \textcolor{#3E6D8E}{\mathbf{x}}}\,\right|

  • Which pixels, if changed a little, would change the logit zcz_c the most?
  • zc\textcolor{#3F7A6B}{z_c}: the logit (score before the softmax) of the predicted class; x\textcolor{#3E6D8E}{\mathbf{x}}: the image
  • s\textcolor{#B23A2E}{\mathbf{s}}: the saliency map, one value per pixel
  • We use the predicted class, to explain what the network did. When the network is wrong, compare with the map for the correct class

Saliency: backpropagate the logit to the pixels

A schematic network, left to right: the petunia photograph, image x; conv blocks with weights theta; feature maps A of layer l; more conv; the last feature maps A of layer L, channels k; class scores z; and at the right, the objective box. Gray arrows carry the forward pass; a red arrow carries the gradient backward. Here the objective is the class score z_c, one cell of z filled red; weights fixed; the red arrow runs back to the pixels; below, the petunia's saliency map, the absolute value of the partial derivative of z_c with respect to x, read the slope at each pixel, no step
  • We put zcz_c at the top and backpropagate to the pixels. The slope at each pixel is the saliency map
  • The next slide works through the chain rule

The chain rule reaches the pixels in one more link

  • We use the backpropagation recipe with zcz_c at the top instead of the loss. A unit's error is now δj=∂zc/∂zj\delta_j = \partial z_c/\partial z_j, computed from the units above, as before
  • Each unit jj of the first hidden layer takes the pixels xk\textcolor{#3E6D8E}{x}_k as its inputs. There, training and saliency use the errors differently:

∂zc∂wjk  =  δj xk⏟training: error×input∂zc∂xk  =  ∑jδj wjk⏟saliency: error×weight, summed over the units pixel k feeds\underbrace{\frac{\partial z_c}{\partial \textcolor{#B5396B}{w}_{jk}} \;=\; \delta_j\,\textcolor{#3E6D8E}{x}_k}_{\text{training: error} \times \text{input}} \qquad\qquad\qquad \underbrace{\frac{\partial z_c}{\partial \textcolor{#3E6D8E}{x}_k} \;=\; \sum_j \delta_j\,\textcolor{#B5396B}{w}_{jk}}_{\text{saliency: error} \times \text{weight, summed over the units pixel } k \text{ feeds}}

  • Treat each pixel as a unit of the input layer. The hidden-unit formula, applied one more time, gives the map
  • On the way down, the errors go through ReLUs whose slope is 0 or 1. The next slide shows why this makes the map noisy

Invisible noise changes the saliency map

Three panels. First, the petunia's saliency map for morning glory, logit 10.69. Second, the saliency map after adding a fixed noise pattern of about 2.6 gray levels to every pixel, logit 11.26: the speckle has rearranged, with bright clusters in new places. Third, the correlation between the map of the noisy image and the photograph's map, against the size of the noise from 0 to 5 gray levels: it falls from 1 to 0.38 at 2.6 gray levels, marked, and to about 0.2 at 5
  • Add noise of about 2.6 gray levels (out of 255) to every pixel. You cannot see it, and the logit barely changes (10.69 → 11.26). Yet the map changes a lot. Its correlation with the original map drops to 0.38
  • Why? Each ReLU has a slope of 0 below threshold and 1 above. A tiny change in the pixel values turns some units on or off, and the gradient jumps

SmoothGrad: average the maps of many noisy copies

The petunia photograph, left, and its SmoothGrad map for morning glory, right: smoother than the saliency map, with the dark center and the petal edges standing out

Computed · 50 noisy copies · same network and photograph

sˉ  =  1n∑i=1n∣ ∂zc∂(x+ϵi) ∣\bar{\mathbf{s}} \;=\; \frac{1}{n}\sum_{i=1}^{n}\left|\,\frac{\partial \textcolor{#3F7A6B}{z_c}}{\partial (\textcolor{#3E6D8E}{\mathbf{x}}+\boldsymbol{\epsilon}_i)}\,\right|

  • SmoothGrad makes 50 noisy copies of the photograph, each with its own noise ϵi\boldsymbol{\epsilon}_i, and averages their saliency maps
  • The jumps differ from one copy to the next, so they average out. What remains is the part of the map that all the copies share
  • You will compute saliency and SmoothGrad maps in the assignment

Saturation: the slope is flat where the detail mattered

The logit for morning glory along a path of images from a blurred copy of the petunia, left, to the photograph, right, with both images shown at the bottom. The logit rises steeply from 2.8 over the first third of the path to a peak near 11.5 at the middle, then declines slightly to 10.69 at the photograph, where a dashed tangent is flat: slope at the photograph 0.07. A double arrow marks the rise of 7.9 between the blurred copy and the photograph

Computed · same network and photograph · blurred copy: Gaussian blur, sd 10 pixels

  • We go step by step from a blurred copy of the photograph to the photograph itself. The logit rises from 2.8 to about 11 in the first third of the way, then stays about flat
  • At the photograph, the slope is only 0.07. The gradient says the detail does not matter, yet adding it raised the logit by 7.9
  • This is saturation. Once the network has enough evidence, more evidence adds nothing, just as a neuron firing at its maximum rate cannot respond more to a stronger stimulus

Integrated gradients: add up the slopes along the path

The petunia photograph, left, and its integrated-gradients map for morning glory, right: credit concentrated on the dark center and along the petal veins and edges, with less speckle than saliency

Computed · baseline: a blurred copy · 32 steps

IG  =  (x−x0) ⊙ 1m∑j=1m∂zc∂x∣x0+jm(x−x0)\textcolor{#B23A2E}{\mathrm{IG}} \;=\; (\textcolor{#3E6D8E}{\mathbf{x}}-\mathbf{x}_0)\,\odot\,\frac{1}{m}\sum_{j=1}^{m}\frac{\partial \textcolor{#3F7A6B}{z_c}}{\partial \mathbf{x}}\Big|_{\mathbf{x}_0+\frac{j}{m}(\textcolor{#3E6D8E}{\mathbf{x}}-\mathbf{x}_0)}

  • Integrated gradients computes the gradient at mm images along the path from a baseline x0\mathbf{x}_0 to the photograph. It multiplies each pixel's average slope by how far that pixel moved, x−x0\mathbf{x}-\mathbf{x}_0. Slope times distance is that pixel's share of the rise
  • Now the steep part of the path counts. The credits of all pixels add up to the whole rise of the logit, 7.9 here
  • You have to choose the baseline. We use a blurred copy here, and the assignment's bonus uses a black image

Grad-CAM: stop at the last feature maps

A schematic network, left to right: the petunia photograph, image x; conv blocks with weights theta; feature maps A of layer l; more conv; the last feature maps A of layer L, channels k; class scores z; and at the right, the objective box. Gray arrows carry the forward pass; a red arrow carries the gradient backward. Here the objective is the class score z_c; the red arrow stops at the last feature maps A of layer L, where each channel k carries a weight alpha k, for example 0.8, 0.1, minus 0.3, 0.5; below, alpha k is the average over positions of the partial derivative of the class score z_c with respect to channel k, and the map for class c is ReLU of the sum over k of alpha k times channel k, drawn as a coarse heat map over the petunia, brightest at its center
  • Each channel kk of the last feature maps gets a weight αk\alpha_k: how much the logit zcz_c rises when that channel is stronger. The weighted sum of the channels is the map. It is coarse, 7 × 7 for ResNet-50, but it lands on the object

Remove a region and run the network again

Five versions of the petunia photograph, each with the probability of morning glory: the photograph, 0.63; a gray square over the dark center, 0.15; the same square inpainted, filled with smooth pink continued from its border, 0.00; the brightest region of the SmoothGrad map, an irregular patch over the center and the petal bases, filled gray, 0.05; the same region inpainted, 0.00. Each removed region is outlined with a dashed line
  • A map claims that some pixels drove the score. We can test that claim as occlusion does. Remove those pixels and run the network again
  • The region can be a square, as in occlusion, or any shape, such as the region a map points to
  • The fill can be gray, as in occlusion, or inpainted. Inpainting fills each pixel by smoothly continuing the colors at the region's border, so no gray shape is added

Test the map against the same shape elsewhere

The petunia photograph, left, with the brightest region of its SmoothGrad map outlined: the dark center and the petal bases above it; right, the same photograph with that region inpainted, filled with smooth pink

Computed · region: the brightest 6% of the SmoothGrad map, grown to 7%

Dot plot of the probability of morning glory: 0.00 after inpainting the map's region, against 30 same-shape controls between 0.50 and 0.82, around the original 0.63
  • If the map is right, removing its region should lower the score more than removing the same shape anywhere else
  • Inpainting the map's region drops the probability from 0.63 to 0.00. As a control, we inpaint the same shape at 30 places that miss the region. The probability stays between 0.50 and 0.82. Only the region the map picked matters
  • This test works for any map. In the assignment, you will measure the share of a map's top pixels that fall on the flower

Test every map: delete pixels in its order

Left, the petunia photograph labelled true petunia, predicted morning glory, p = 0.633. Right, deletion curves: the probability of morning glory against the fraction of pixels replaced by a blurred copy, pixels removed in the order each map ranks them, 10 steps. Occlusion falls fastest, area 0.04, then Grad-CAM 0.08, saliency 0.18 and integrated gradients 0.21; the dashed random map stays high for most of the way, area 0.58

  • Deletion removes pixels in the order the map ranks them, 10% at a time. The area under the curve scores the map. With a good map, the probability falls fast
  • Random map 0.58, occlusion 0.04, Grad-CAM 0.08, saliency 0.18, integrated gradients 0.21

What does a single unit respond to?

Find the image that drives one unit hardest

Feature visualization: one channel at the top

A schematic network, left to right: the petunia photograph, image x; conv blocks with weights theta; feature maps A of layer l; more conv; the last feature maps A of layer L, channels k; class scores z; and at the right, the objective box. Gray arrows carry the forward pass; a red arrow carries the gradient backward. Here the objective box is wired to the feature maps A of layer l, and everything after them is grayed out; one channel k is outlined red; objective: one channel's mean activation, one over H_l W_l times the sum over i, j of A l, i, j, k; weights fixed; the red arrow runs back to the pixels; below, a real feature visualization of a ResNet-50 channel, and x gets x plus eta times the gradient, many steps, start from noise
  • At the top, we now put one channel's mean activation. We start from noise and take many steps up its gradient. The result is the image that drives the channel most

Feature visualization: optimize the image

We keep the weights fixed and change the image until one channel kk of layer ll, averaged over its Hl×WlH_l \times W_l positions, responds as strongly as possible:

x⋆  =  arg⁡max⁡x  1HlWl∑i,jAi,j,k[l](x)  −  penalty(x)\textcolor{#3E6D8E}{\mathbf{x}}^\star \;=\; \arg\max_{\textcolor{#3E6D8E}{\mathbf{x}}}\; \textcolor{#3F7A6B}{\frac{1}{H_l W_l}\sum_{i,j} A^{[l]}_{i,j,k}(\mathbf{x})} \;-\; \text{penalty}(\textcolor{#3E6D8E}{\mathbf{x}})

  • arg max means the image x\mathbf{x} that makes the expression largest. Training walked downhill on the loss. Here, you walk uphill on the channel's mean activation
  • Without the penalty, the result is high-frequency noise that drives the channel but means nothing to you, much like the adversarial image from the start of today. The penalty keeps the image natural-looking
  • This asks what channel kk responds to across all images, instead of explaining one decision. The image found is the channel's maximally exciting image (MEI)

AlexNet: oriented edges, then textures, then object parts

AlexNet feature visualizations in three groups of four. First layer, the filters: oriented stripes and color blobs. Middle layer, conv3: repeating textures of dots and cells. Last layer, conv5: object-like parts, a coral-like surface with holes, rows of eye-like spots, shells
  • The first layer has oriented edges and color blobs, like V1 simple cells. The middle layers respond to repeating textures, and the last layers to object-like parts

ResNet-50: edges, textures, then sharper parts

ResNet-50 feature visualizations in three groups of four. First layer, the 7 by 7 filters: color-opponent blobs and an oriented blue-orange edge. Middle, layer3: textures and arch-like repeated shapes. Last, layer4: screws and bolts, building facades, traffic seen from above
  • In the last block, the parts are recognizable: screws, facades, traffic. Nobody told the network about any of them. This is the ResNet-50 you use in the assignment

ViT-B/16: no oriented filters in the first layer

ViT-B/16 feature visualizations in three groups of four. First layer, the 16 by 16 patch filters: noisy, colored, with no oriented stripes. Middle, block 5: repeating scale and wave textures. Last, block 11: diffuse, speckled patterns with small dark marks, no clear object parts
  • A vision transformer cuts the image into 16 × 16 patches. Unlike a convolutional network, its first-layer filters show no preferred orientation

DINOv2, trained without labels: textures and parts too

DINOv2 ViT-B/14 feature visualizations in three groups of four. First layer, the 14 by 14 patch filters: noisy, no oriented stripes. Middle, block 5: woven, ridged and pebbled textures. Last, block 11: dense surfaces with eye-like spots, fur-like and ridged textures
  • DINOv2 is self-supervised. Without labels, it learns to give matching outputs for two views of the same image. Its first layer has patch filters like ViT's, and its top layers still show textures and parts

ResNet-50's last block: MEIs and top photographs

Three ResNet-50 layer4 units, one per row: the unit's MaCo feature visualization, then its six most strongly activating ImageNet validation images. Unit 43: a helmet, a triceratops skull, a thatched roof, a burrito, a thatched cottage, axes. Unit 179: a bell, screws, a perforated bone, a bolt, a birdhouse, a shark. Unit 309: sleeping bags, volcanoes and mountain peaks, a tent, a yurt

  • Each row is one unit. The MEI is synthesized and shows only what the unit prefers, but it is hard to read. The top photographs are the six ImageNet photographs that drive the unit hardest, real but cluttered. Look at both together

A shortcut: "tench" includes the fingers holding it

Two concepts of ResNet-50 for the class tench, from ObjectLENS. Left, concept 2: a synthesized image of fish bodies with pink fingers wrapped around them, beside three ImageNet crops that drive it, each a hand gripping a fish. Right, concept 6: a synthesized image of fish heads, beside three crops of tench heads and faces

  • ObjectLENS, a tool from our lab, splits what the last block does for one class into a few concepts. It visualizes each one and shows the photographs that drive it
  • Most ImageNet tench are trophy shots, held up by an angler. The network learned the hands as part of the fish, a shortcut: a cue that predicts the label without being the object
  • Try at home: all 1,000 classes at serre-lab.github.io/Lens

A Spider-Man neuron, after the Halle Berry neuron

Seven image tiles that drive one unit of CLIP: photographs of two people in Spider-Man costumes and of a real spider in its web; a pencil drawing of the mask, a Spider-Man sticker and a drawing of a spider; and a sign reading The Amazing Spider-Man
  • Quiroga et al. (2005) recorded a neuron in a patient's medial temporal lobe that fired for photographs of Halle Berry, a drawing of her, and her written name. It responds to the person, whatever the picture
  • CLIP learns from 400 million photographs with their captions. One of its units responds to Spider-Man in photographs, drawings and the written name, and to spiders. It was found with today's tools, feature visualization and top-activating images
  • Because CLIP reads text, text can fool it. An apple with a note that says "iPod" is classified as an iPod, a typographic attack

A Golden Gate Bridge feature in a language model

Example inputs that activate the Golden Gate Bridge feature of Claude 3 Sonnet, from Templeton et al. (2024): three photographs of the bridge, at sunset, from below and with the Space Shuttle flying over it, and four lines of text in which the words Golden Gate Bridge are highlighted
  • Templeton et al. (2024) found millions of features in the language model Claude 3 Sonnet. Each feature is a direction in the model's activity. One responds to the Golden Gate Bridge in photographs, in English and in other languages
  • If you turn that feature up, the model brings the bridge into every answer. In any network, we can ask what drives a unit and what happens when we drive it
  • Try at home: browse the features of open language models at neuronpedia.org

Is a network a good model of the ventral stream?

Score a network's blocks against V1, V4 and IT recordings, with tools you already have

Scoring a network against the brain: tools you know

Sixteen photographs of everyday objects from the THINGS database, each labeled with its concept, alligator to boat in alphabetical order, for example an axe in snow, a banana, a beaver and a sailing boat

TVSD (Papale et al. 2025): 16 of the 100 test photographs

  • A readout is a ridge regression from a block's units to one site's responses, fit on training photographs and scored on new ones
  • The noise ceiling divides each site's score by how well the site predicts itself across repeats, so 1 is the best possible
  • A block's linear predictivity is the median score over an area's sites. RSA compares dissimilarity matrices, with no weights to fit
  • Data, as in the assignment: TVSD, about 22,000 THINGS photographs, sites in V1, V4 and IT of two monkeys

Later blocks predict later areas

Median noise-corrected correlation of ridge encoding models against the 16 blocks of an ImageNet ResNet-50, for TVSD V1 in blue, V4 in green and IT in magenta; monkey F solid, monkey N dashed; light bootstrap bands; a dot marks each curve's peak. V4 peaks at layer3.3 in both monkeys, about 0.68 and 0.71; IT peaks later, at layer4.0 in F, 0.76, and layer3.4 in N, 0.81; V1 stays near 0.64 and peaks at different blocks in the two monkeys; every curve falls in the last blocks

  • We fit a readout from every block of an ImageNet ResNet-50 to every site. The assignment's last part does this for the last block only. V4 peaks at layer3, and IT keeps rising after V4 has levelled off. All areas fall in the last blocks

Better AI has not meant better models of the brain

Past human-level accuracy, better ImageNet networks are not better models of IT, and the recordings disagree about what is

Past human-level accuracy, the trend reverses

Scatter of neural alignment, 0.2 to 1.0, against ImageNet multi-label accuracy, 0.5 to 1.0, for several hundred models colored by family: CNNs trained on ImageNet-1K in red, adversarially trained CNNs in yellow, convolutional vision transformers in green, vision transformers in blue, with internet-scale and vision-and-language variants in lighter shades; a gray band marks human accuracy near 0.93 to 0.97. Two dashed lines: alignment rises with accuracy up to about 0.9, then falls; purple stars mark recent models SigLIP-2, DINOv3 ViT-L and ViT-H, I-JEPA, vJEPA2 and EVA-02, the most accurate of which sit lowest

  • Each point is a network, scored on how well its features predict monkey IT. Up to about human accuracy, the fit rises. Beyond that, the most accurate networks are worse models of IT

Four datasets, one cohort: the picture in 2026

Four scatter plots of alignment score against ImageNet multilabel accuracy for 363 classifiers, CNNs as blue dots, transformers orange, self-supervised and image-text models as open circles, adversarially trained models as triangles, each with a dashed noise ceiling. Arcaro IT: rank correlation plus 0.37 overall, minus 0.38 among the 84 models above 92.7 percent. Triple-N IT: minus 0.08 overall, plus 0.20 at the top. TVSD IT: plus 0.73 overall, plus 0.65 at the top. Human feature use: minus 0.77 overall, minus 0.37 at the top

Reproduced from Serre, Annu Rev Vis Sci (in submission), Fig. 4 (draft); Triple-N and TVSD scores computed for the review

  • 363 networks, four targets. For IT, we use each network's best-fitting layer. Human feature use is the agreement with the regions people paint as important in ClickMe, a web game
  • Arcaro IT and ClickMe turn down past about 93% accuracy. TVSD still rises (rank correlation ρ = +0.73). Triple-N, a newer IT dataset, is flat (ρ = −0.08)
  • The answer depends on which recordings you use. To discuss: if higher accuracy no longer gives a better fit to the brain, what would?
Serre Annu Rev Vis Sci (in submission) · Linsley, Feng & Serre Trends Cogn Sci 2026 · Li et al. Nat Neurosci 2026

Final projects: ask the models

  • Everything today is an experiment on a model: a recording, a readout, a designed image. The data are on hand: TVSD, THINGS behavior, ClickMe maps, the three ResNet-50s you already have, the Oscar GPUs
  • One experiment toward a neural law: a quantitative relation between what a model was given (images, objective, architecture, budget) and what it predicts (sites, behavior)
  • Seeds: hold the architecture constant and vary the objective (labels, DINO, image–text): which predicts TVSD or Triple-N better, and at which block? · Take one published comparison and do it better: matched layers, noise ceilings, the sites a model misses · Make two equally scoring models disagree: synthesized images, MEIs, controversial stimuli, then test which is right · Train with a human signal (ClickMe) and ask whether IT prediction follows
  • A careful redo counts. So does a negative result with the right controls
Serre Annu Rev Vis Sci (in submission) · Golan, Raju & Kriegeskorte PNAS 2020 · Fel et al. NeurIPS 2022

Recap

  • Backpropagation has four uses. Training changes the weights. With the weights fixed, the gradient with respect to the pixels can change the answer (adversarial example), explain it (attribution map), or synthesize the image a unit prefers (feature visualization, MEI)
  • An attribution map is a hypothesis. Test it by removing its region and comparing with the same shape elsewhere, or by deleting pixels in its order
  • In convolutional networks, feature visualization shows edges, then textures, then parts. A transformer's first layer has no oriented filters. These tools also find shortcuts (tench and fingers), units that respond to one concept in photographs, drawings and text (Spider-Man) and features in language models
  • Linear predictivity fits a ridge readout from a network block to each recorded site, scored on held-out images relative to the site's reliability. Deeper blocks predict later areas, V4 before IT
  • Better AI has not meant better models of the brain. Past about human accuracy, more accurate networks do not reliably predict IT better, and the datasets disagree

Further reading for the research trail

Optional: a few entries over the semester; any paper from any lecture qualifies.

Minute paper

Submit on Canvas → Minute papers → Minute Paper 14 (access code read out in class).

The minute paper asks you to recall, in your own words, what you just heard. Retrieving it is what strengthens the memory; copying a slide does not. It is not a quiz.

A trend I have noticed: papers that discuss slides I never reached, and papers written while I was still lecturing. Both miss the point. From today, a paper about material not covered in class, or written before the class is over, loses credit.

Write three brief points:

  1. One idea from today, restated in your own words
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Going further

Decoding: a readout from brain to stimulus

Eight polar plots, one per grating orientation shown (horizontal, then in steps of 22.5 degrees). In each, a black line marks the orientation shown and a gray outline shows how often the readout chose each orientation; every outline is elongated along the black line. Label: participant S2, V1 and V2, 400 voxels

Reproduced from Kamitani & Tong (2005), Fig. 2a

  • Decoding (MVPA): a linear readout run from brain to stimulus. Kamitani & Tong (2005): gratings at 8 orientations. Each voxel (a small cube of cortex) prefers one orientation only slightly; a readout over 400 voxels picks it well above chance
  • The information is in the pattern across voxels

Decoding dreams

  • Decoding dreams, from Horikawa et al. (2013): readouts trained on fMRI while people viewed photographs were applied to fMRI recorded in sleep, just before each awakening. They predicted the object categories people then reported from their dreams above chance
  • A readout can only choose among the labels it was trained on. This tests which visual representations dreaming reuses. It is not mind reading

- Every method in the first half of today uses backpropagation from the training lectures with one change: the gradient is taken with respect to the image's pixel values, and the weights stay fixed. Adversarial images and feature visualization then step the pixels; attribution maps only read the gradient

- Training, from the backpropagation lectures: the gradient lists one slope per weight, and gradient descent takes a small step against it, again and again - The figure is the smallest network there is: one weight w times one input x, scored with the logistic loss for the correct class. Training (magenta) starts from a poor weight, w = 0.15, and takes eight gradient steps on w with x fixed, to w = 2.62 (loss 0.63 to 0.12). The attack (blue) starts from the trained network and takes five fixed-size steps on x along the sign of the gradient, up the loss, with w fixed (loss 0.12 to 1.98). The dashed line, wx = 0, is the decision boundary; below it the answer is wrong - With this loss there is no finite best weight: the loss keeps falling as w grows, so "trained" here means a low loss, and no finite weight is the best one - Up the gradient is the direction in which the loss rises fastest for a small step, so the attack maximizes the loss over the image, within a budget on how much each pixel may change, while training minimizes it over the weights. Adversarial training combines the two: minimize over the weights the loss of the worst image within the budget - Why that direction is also the shortest way to the boundary: for a linear readout the loss depends on x only through w·x, so the gradient with respect to x is along w, the normal to the boundary, and the boundary itself is a contour of the loss. For a deep network this holds only locally, step by step; the figure has a single input dimension, so the only choice is up or down - Weights and pixels enter the same chain of multiplications, additions and ReLUs, and the chain rule runs back to either. Training stops at the weights; the methods today run one step further, to the pixels - 25,557,032 weights in torchvision's ResNet-50; the input is a 224 × 224 color image

- Every row computes a gradient by backpropagation. The rows differ in what is differentiated, with respect to what, and what is done with the result. Training goes down the loss by changing the weights, starting from random ones. The other three never change a weight. Each asks the network a different question: what breaks it, which pixels it used, what a unit responds to - An adversarial example goes up the loss of the correct answer, within a budget small enough that people cannot see the change: next slide - An attribution map takes no step and changes nothing: the gradient at the photograph is itself the map. It says which pixels would change the class score most if they changed a little - Feature visualization does not start from a photograph: it starts from noise and climbs one channel's mean activation for many steps, so the result is a synthesized image. Neuroscientists run the same optimization on a model of one neuron and call the result its maximally exciting image (MEI)

- The network alternates between conv blocks, which hold the weights, and their outputs, the feature maps. Only two stacks are drawn and labeled, $A^{[l]}$ after a middle block and $A^{[L]}$ after the last, because later slides use them: Grad-CAM stops the gradient at the last layer's feature maps A^[L], and feature visualization starts it at an intermediate layer's A^[l]. Training uses neither; it needs only the weights - The objective is the loss ℓ(z, y); the gradient runs back and stops at every weight. This is backpropagation as in the training lectures, drawn for a convolutional network: one forward pass computes the loss, one backward pass gives a slope for every weight, and gradient descent changes the weights a little - On the slides that follow the weights are locked. The backward pass is the same; it runs one step further, to the pixels, or stops earlier, at a layer

- In the schematic the change is drawn as random signs. The next slide shows the real one: the sign of the gradient for a panda - x + ε · sign(∇ₓ ℓ) is the fast gradient sign method of Goodfellow et al. (2015). The sign comes from the budget: the change must stay invisible, so each pixel may move by at most ε. Under that rule the first-order best step moves every pixel by the full ε, up if its slope is positive and down if negative; a steep slope and a shallow one get the same ε. With a budget on the total length of the change instead, the step would follow the gradient itself, steep pixels moving more. The demo after it repeats the step ten times within the same budget

- Goodfellow, Shlens & Szegedy (2015), Explaining and harnessing adversarial examples. The fast gradient sign method: x + ε · sign(∇ₓ loss), with ε = 0.007 of the pixel range. Later today a gradient with respect to the pixels, of the logit instead of the loss, is the saliency map: there it explains the answer instead of changing it - The budget is a norm of the change δ = x' − x. L∞: the largest change to any one pixel, the panda's budget and the demo's; L2: the length of the whole change vector, the assignment's. The two budgets give different worst images, and a network robust under one need not be robust under the other - Robustness is read from the budget: the smallest ‖δ‖ that flips the answer, or the accuracy that survives attacks within a fixed ε. Random noise of the same size almost never flips the answer; the gradient finds the one direction that does - People are not fooled by perturbations this small; with a very brief presentation, larger perturbations made for many networks at once can bias human choices slightly (Elsayed et al. 2018)

- Madry et al. (2018), Towards deep learning models resistant to adversarial attacks; Salman et al. (2020), Do adversarially robust ImageNet models transfer better? The assignment's robust ResNet-50 is theirs - The inner max is solved approximately, by a few gradient steps up the loss on the pixels, each followed by a projection back into the budget; the outer min is a gradient step down the loss on the weights. Backpropagation used twice per step, once to the pixels and once to the weights - The attacks are regenerated at every step for the network as it is now, so they keep targeting its current weak spots - Robust networks' gradients look more like the object, and their representations predict some neural data better; the assignment's last part asks whether the robust ResNet-50 predicts IT better than the standard one

- Each setting is precomputed: ten gradient steps up the loss of the network's own answer on the original, each pixel kept within the attack size of its original value (projected gradient descent with an L-infinity budget). The probabilities are the network's on the attacked image - Standard ImageNet ResNet-50 with the flower readout: the petunia it calls morning glory (0.63) becomes bougainvillea (1.00) at half a gray level out of 255; the passion flower and the rose flip as fast - Robust ResNet-50 (Salman et al. 2020, adversarially trained) with its own flower readout, the one in the assignment: it calls the petunia petunia (0.91) and holds it up to 2 out of 255 (0.59); the passion flower holds to 6 out of 255 - Switch networks at the same attack size: the change for the robust network is not noise: amplified, it shows edges and petal shapes, because pushing a robust network's answer takes changes to what the object looks like. This is why robust networks' gradients make cleaner attribution maps and feature visualizations - The demo is on the course site with the other demos

- These slides ask which pixels drove one network's decision, the question of the assignment's third part - The gradient that built the adversarial example, read at the photograph instead of stepped along, explains a decision - Occlusion first, then the gradient at the pixels (saliency), why that map is noisy and why it misses detail, Grad-CAM, and two tests of any map

- Reproduced from Zeiler & Fergus (2014), Fig. 7, columns (a) and (d), laid out in a row, color bars dropped (red is high, blue low): in each pair, left, the input with the gray square at one position; right, the probability of the correct class (Pomeranian, car wheel, Afghan hound) with the square centered at each position - Covering the Pomeranian's face drops the probability of "Pomeranian" sharply; covering other parts of the image barely changes it. For the third image, covering the dog matters and covering the man's face does not, although the man's face drove the strongest feature map in the network's top layer - An occlusion map asks what a decision depends on, by changing the input. A unit's strongest feature map asks what a unit responds to. The two need not agree - Why the caveat matters: the network never saw gray squares in training, so the square can lower the score on its own - Two checks. Change the fill: a blurred copy, noise, or the region filled in from its surroundings (inpainting, a few slides on). If the map's bright region stays put across fills and square sizes, the drop comes from what is hidden. And place the same square on a region the map calls unimportant: if the score drops there too, the square itself is doing the work - Gradient-based attribution, after the demo, asks the same question without rerunning the network for every position

- Sweep: the square visits every position and draws the map for the selected class. Unlike the previous slide, the map here is the drop in probability (clean minus covered), so bright marks the regions that matter - Clean image: English springer 0.625, hen 0.018, Irish water spaniel 0.130, Sussex spaniel 0.065, miniature poodle 0.030 (ImageNet ResNet-50; 1,795 reruns precomputed: 16, 32 and 64 px squares at every 8-pixel position) - A 32 px square on the dog's face (row 40, column 136) sends springer 0.625 → 0.000 while hen rises to 0.371 and miniature poodle to 0.388; a 64 px square there gives hen 0.835 - A 32 px square on the hen's head (row 128, column 40) barely moves springer (0.587): the decision does not depend on the hen - The hen map's own peak is only 0.018, so on the fixed 0–0.625 scale it is flat: read a map together with the score it explains - The demo is on the course site with the other demos

- In the lecture on learning the weights, backpropagation computed the gradient of the loss with respect to the weights. Here the same chain rule gives the gradient of one logit with respect to the pixels; the weights stay fixed - For a linear neuron this map would be its receptive field, the weights; for the network it is the class unit's receptive field measured at this one photograph, and it changes from photograph to photograph, unlike a map from reverse correlation, which averages over many stimuli - The logit is used instead of the probability, so the map does not also reflect the scores of the other classes through the softmax - Absolute value: a pixel matters whether brightening or darkening it raises the score - In practice: one forward pass, one backward pass, then the largest of the three color channels at each pixel. In PyTorch, `x.requires_grad_(True)`, then `logits[0, c].backward()`, and the map is `x.grad.abs()`

- Training's backpropagation with two changes: the logit z_c at the top instead of the loss, and the pixels at the bottom instead of the weights. The map is large where a small change of that pixel moves z_c most - Why z_c and not the loss: the loss also depends on the label and on every other class through the softmax; the logit z_c asks about one class only - Which class c: by default, the predicted class, because the map should explain what the network did; it needs no label, so it works on any image. The assignment does this. For a mistake, also compute the map for the correct class and compare the two: the predicted-class map shows the evidence that drove the wrong answer, the correct-class map shows whether the network found evidence for the right one, and where. The petunia here is such a mistake: the map is for morning glory, the network's answer

- In the backpropagation lecture the error of a unit was δ = ∂ℓ/∂z, the slope of the loss with respect to its weighted sum, and the hidden-unit formula computed it from the errors of the units above: multiply each by the weight on the connection and by the slope of the activation, then add. Here the quantity at the top is the logit z_c instead of the loss, so the output layer's error is simply 1 for unit c and 0 for the others; everything below is the same recipe - Training uses the first hidden layer's errors once more to get the weight gradient, error times input, and stops. Saliency uses the same errors to get the pixel gradient: pixel k enters the weighted sum of every unit j of the first hidden layer, the first convolution, with weight w_jk, so its slope is the sum of δ_j w_jk over those units. A pixel has no activation function of its own, so no g′ term appears - In a convolutional layer the sum over j runs over the units whose receptive field contains the pixel, and w_jk is the filter weight at that offset: the backward pass through a convolution is a convolution with the flipped filter - Autograd does this bookkeeping: `x.requires_grad_(True)`, a forward pass, `logits[0, c].backward()`, and `x.grad` holds the map. Training reads the weights' `.grad` instead; one call to `.backward()` fills both

- Computed for the course: the flower network and the petunia of the previous slides, for the class it predicts. One fixed Gaussian noise pattern (sd 0.1 in [0, 1] units, the noise SmoothGrad uses) is scaled down and added to the photograph; the horizontal axis gives the typical change of each pixel's value, out of 255 - A saliency map is the gradient at one image. ResNet-50 in evaluation mode is built from ReLUs, max-pooling, averages and fixed rescalings, so its logit is a piecewise-linear function of the pixel values: flat pieces glued together, with a different slope on each piece. Nearby images sit on different pieces, so their gradients differ, while the logit itself changes smoothly - So a single saliency map mixes the pattern that matters with jumps that would change at the next image. The fix on the next slide averages the maps of many nearby images

- The noise here is much larger than on the previous slide: sd 0.1 in [0, 1] units, about 25 gray levels, as in the assignment. Each copy lands on a different linear piece of the network, so averaging over copies averages over the jumps. It is not a blur of one map: each copy's map is computed by its own backward pass - SmoothGrad: Smilkov et al. (2017); the assignment uses 50 noisy copies, noise 0.1 - The maps on these slides are raw; display tools often smooth them

- Each point of the curve is the image (1 − a) × blurred copy + a × photograph, for a from 0 to 1: every pixel's value moves in a straight line from its blurred value to its value in the photograph - The gradient at the photograph answers only: if the pixel values change a little from here, how much does the logit change? On the flat part, almost not at all. It cannot say that the detail, compared with no detail, raised the logit - Saliency and SmoothGrad both read slopes near the photograph, so both share this limit. The next method uses the whole path

- x − x₀ is fixed along the path, but it is not one number: it is a whole image, one value per pixel, the difference between the photograph and the baseline. It does two things. It turns a slope into a change of the logit, as slope × distance does on a hill; adding these over all pixels gives the whole rise from the baseline to the photograph. And a pixel that is the same in the photograph and the baseline gets no credit, however steep its slope - The path is the one of the previous slide. Each pixel's credit is its average slope along the path times the change of its value (⊙: pixel by pixel). Summed over all pixels, the credit equals the logit at the photograph minus the logit at the baseline, 10.69 − 2.83 = 7.86, up to the error of using m = 32 steps - The baseline says what "no evidence" means. A black image is the original paper's and Xplique's default, used in the assignment's bonus; a blurred copy keeps the colors and removes the detail, so the map credits the detail - Integrated gradients: Sundararajan, Taly & Yan (2017), ICML

- The gradient stops at the last feature maps; each channel's weight α_k is the slope of z_c with respect to channel k, averaged over positions; the map is the weighted sum of channels with negatives set to zero, 7 × 7 for ResNet-50, then enlarged to the image. Each channel of the last convolutional layer is a map of where one feature is present; the gradient says how much z_c rises when that feature is stronger. Weight each map by that sensitivity, add them up, and keep the positive part: the regions whose features push z_c up. Two indices: c is the class being explained, k runs over the channels of the last feature maps - The α values in the figure are illustrative, not computed. The assignment's bonus computes Grad-CAM for the flower network - Selvaraju et al. (2017), Grad-CAM: visual explanations from deep networks via gradient-based localization, ICCV

- Computed for the course: the flower network, the petunia, the probability of morning glory. The map's region is the brightest 6% of the SmoothGrad map, its largest connected patch, grown to 7% of the image. The square has the same area and is centered on the same patch, so it covers mostly the same pixels - All four removals take the dark center, and all four drop the probability, from 0.63 to between 0.15 and 0.00. Here inpainting drops it further than gray: the gray patch keeps a dark, roughly central blob, which the network may still partly read as the flower's center - Inpainting: OpenCV's Telea method. It fills the region from its border inward, each new pixel a weighted average of the known pixels near it - A drop alone does not test the map. The next slide adds the control

- Region: the brightest 6% of the SmoothGrad map's pixels, its largest connected patch, grown slightly: 7% of the image - Removing it drops p(morning glory) from 0.633 to 0.001; the 30 same-shape controls leave it between 0.50 and 0.82, median 0.67, none at or below the map region's value. Here the map and the intervention agree - Why a control: any edit makes the image less natural and can lower the score by itself. The control removes the same amount of image, of the same shape and size, placed where it misses the map's region - Same logic as a lesion or a muscimol inactivation with a sham control: remove a part, compare with removing an equal part elsewhere. As with a lesion, the edit can add something new (here a smooth pink patch), which is what the control is for - A smoother map is not a more accurate one: SmoothGrad looks cleaner than saliency, but only a test like this one says whether its region is the one the network uses - The assignment uses a simpler test: the share of the map's top pixels on the flower's segmentation mask, against a map that ignores the network

- Computed for the course: the same network and photograph. The previous test removed one region. Deletion tests the whole ranking: pixels go in the order the map gives them, 10% at a time, each replaced by the blurred copy of the photograph, and the class probability is recorded after every step. Any map can be scored this way, so the maps can be compared on one photograph - The area under the curve summarizes it: a map that finds the pixels the network uses drops the probability within the first few percent and gets a small area; a random map removes pixels in no useful order and gets a large one - Blurring 10% of the pixels can also raise the probability (the random and integrated-gradients curves rise at the first step): the blurred copy removes detail the network read against morning glory. Only the fall over the whole curve counts - Insertion is the mirror test: start from the blurred copy and put pixels back in the map's order; a good map restores the score fast - The rankings depend on the photograph: do not assume this order holds elsewhere. Occlusion's map is itself made by removing regions, which is what deletion scores, so it has a head start

- The same tool, run on a model of an IT neuron, makes the images of the assignment's last part

- The layers above layer l play no part. They are grayed out because the objective does not depend on them: the backward pass starts at layer l - The result panel shows a real visualization: channel 959 of ResNet-50's layer3, made with MaCo like the images that follow

- Maximally exciting image, MEI: the name neuroscientists use for the image that drives a neuron, or a model of one, hardest. The assignment's last part computes MEIs for models of IT sites. Bashivan et al. (2019) showed such images to a monkey and drove V4 sites past every natural image; it is on the reading list - Tanaka's reduction method searched by hand while recording; here the gradient does the search. A neuron gives no gradient, so neuroscientists fit a model of the neuron and optimize the model (Walker et al. 2019, Bashivan et al. 2019), or search without gradients, by evolving images from the neuron's firing (Ponce et al. 2019) - The fourth row of the uses table: the gradient to the pixels, many steps, to maximize one channel - Typical penalties: small random shifts of the image at every step (jitter), blurring, and penalizing fine high-frequency detail - The images on the next slide use MaCo (Fel et al. 2023), which keeps the image's spectrum at that of natural images and optimizes only the phase - Unit: one channel of a layer, its activation averaged over positions

- Computed for the course with MaCo (Fel et al. 2023), four units per layer; first layer: the filters themselves, not optimized images - AlexNet (Krizhevsky et al. 2012), torchvision ImageNet weights. Units: the four highest-scoring of 16 units chosen for their variance over 500 ImageNet photographs, not for any category, so no unit was picked because it looks like a face or a car - First layer: the 11 × 11 filters themselves (weights), shown with zero as mid-gray. Middle and last: MaCo images, each the input that drives one channel's mean activation hardest - A hierarchy like the one from V1 to IT: small oriented features first, then textures, then larger parts.

- Computed for the course with MaCo (Fel et al. 2023), four units per layer; first layer: the filters themselves, not optimized images - ResNet-50 (He et al. 2016), torchvision ImageNet weights, the encoder of the flower network and of the assignment. Units chosen as for AlexNet; the last-block units are 179, 1975, 1352 and 1839; unit 179 comes back with its top photographs after the two transformer slides - First layer: the 7 × 7 filters. At this size they are blurry, but color-opponent centers and an oriented edge are visible - Face units: none of these sampled units is a face unit. ImageNet has few face categories; studies that find face-selective units search for them, for example by the images that drive each unit

- Computed for the course with MaCo (Fel et al. 2023), four units per layer; first layer: the filters themselves, not optimized images - ViT-B/16 (Dosovitskiy et al. 2021), supervised on ImageNet. A vision transformer's first layer cuts the image into non-overlapping 16 × 16 patches, a grid of 14 × 14 for a 224-pixel image, and turns each patch into a token: one linear projection maps the patch's 16 × 16 × 3 = 768 pixel values to 768 numbers. Attention then mixes tokens across the whole image from the first block on - Why no orientation: in a convolutional network, each first-layer filter is a small window slid over every position, so a unit has to be a local detector, and training on natural images makes it an oriented edge detector, as V1 cells are. A patch projection is not slid over overlapping positions and has no built-in preference for neighboring pixels: each output channel mixes all 256 pixels of a patch. Orientation is spread over many channels instead of being the preferred direction of one; the principal components of the projections do look like smooth, Gabor-like basis functions (Dosovitskiy et al. 2021, Fig. 7) - Position is not built in either: a learned position embedding is added to each token - The last block's images are less part-like than ResNet-50's: each token can draw on the whole image, so a unit need not respond to one localized part

- Computed for the course with MaCo (Fel et al. 2023), four units per layer; first layer: the filters themselves, not optimized images - DINOv2 ViT-B/14 (Oquab et al. 2024): a vision transformer with 14 × 14 patches, trained without labels to give matching representations to different crops of the same image - Its first layer has the same patch structure as ViT-B/16 and the same lack of oriented filters (previous slide) - Labels are not what makes textures and parts appear: a network trained only to recognize that two crops come from one image learns them too. The assignment's other self-supervised network, DINO ResNet-50, is a convolutional network trained the same way

- Three units of layer4 of the ImageNet ResNet-50. Unit 309: conical shapes (tents, volcanoes, yurts); unit 179's photographs (a bell, screws, a shark) share no single label: a unit can respond to several unrelated things - Top-activating images are the classic method of Zeiler & Fergus (2014) and Zhou et al. (2015): units for faces, text, wheels and fur appear in networks never told to find them - Top images: run the whole dataset through the network and keep the images that drive the unit hardest - ResNet-50 (torchvision weights), layer4, six top images from 50,000 ImageNet validation images

- How the concepts are found (CRAFT, Fel et al. 2023): take the last block's activations on crops of the class's photographs and factorize them into a few non-negative parts; each part is a direction in the block's activity, a concept, ranked by its contribution to the class score. Each concept is then visualized with MaCo and shown with the crops that activate it most - A shortcut (Geirhos et al. 2020): a feature that predicts the label in the training photographs without being the object. The network can be right for the wrong reason, and an accuracy score does not show it. A test: a tench photographed without hands, or a hand without a fish - Concept 6 is fish heads, the object itself. The site's importance percentages are each concept's share of the class score, not a causal test - Geirhos, Jacobsen, Michaelis, Zemel, Brendel, Bethge & Wichmann (2020), Shortcut learning in deep neural networks, Nature Machine Intelligence

- Quiroga, Reddy, Kreiman, Koch & Fried (2005), Invariant visual representation by single neurons in the human brain: recordings in epilepsy patients; one neuron fired for Halle Berry across photographs, a drawing and the letter string, another for Jennifer Aniston - Goh, Cammarata, Voss, Carter, Petrov, Schubert, Radford & Olah (2021), Multimodal neurons in artificial neural networks: in CLIP (Radford et al. 2021), a network trained to match photographs with their captions, many units in the last layers respond to a concept across photographs, drawings and text: people, places, emotions, holidays. The tiles are the article's own examples - Typographic attack: a Granny Smith apple photographed with a handwritten note "iPod" is classified as an iPod with high confidence; a photograph with the word "pizza" written on it is called a pizza. The units that read the concept also read its name

- Templeton, Conerly, Marcus, Lindsey, Bricken, Chen, Pearce, Citro, Ameisen, Jones, Cunningham, Turner, McDougall, MacDiarmid, Freeman, Sumers, Rees, Batson, Jermyn, Carter, Henighan & Olah (2024), Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet - A feature here is not one unit. Single units of a language model respond to many unrelated things, so the authors fit a dictionary of directions (a sparse autoencoder) to the activity of one layer and read each direction's top-activating inputs, the language analog of the top photographs of ResNet-50's units. The figure shows the inputs that activate the Golden Gate Bridge feature most strongly: photographs of the bridge and text naming it, in several phrasings - Driving the feature: add its direction to the activity, at several times its usual strength, and the model describes itself as the bridge and brings the bridge into unrelated answers. Served to the public for one day in May 2024 as Golden Gate Claude - No gradient here: the feature's activity is read directly and set directly. The questions are those of feature visualization and of the maximally exciting image: what drives it, and what it does to the output - The networks' layers form a hierarchy like the one from V1 to IT. Whether the blocks also match the areas is the question of the next section

- So far today: which pixels one decision used, and the image a unit likes most. Now the brain: score a network's blocks against V1, V4 and IT recordings with the readout and noise ceiling you already know

- Every tool on this slide came earlier in the course: the encoding model and ridge regression in the vision lecture, the noise ceiling and split-half reliability in the baselines lecture, RSA in the MDS lecture. Nothing new here, only the data - A recording site is one electrode, picking up the summed spiking of the few neurons near its tip (multi-unit activity) - Score used on the next slides: the Pearson correlation between a site's predicted and recorded responses on the 100 test photographs, divided by the square root of the site's split-half reliability (Spearman–Brown corrected): the noise-corrected correlation. The square root, because a reliability is already a fraction of variance - The THINGS Ventral Stream Spiking Dataset (Papale et al. 2025): monkeys F and N looked at 22,248 photographs of 1,854 object concepts from THINGS (Hebart et al. 2019), each seen once, and 100 test photographs seen 30 times, which give each site its reliability. Each site's readout is fit on 5,000 of the single-presentation photographs - Each site's readout: ridge regression from a block's features (each feature map average-pooled to 2 × 2, then flattened), λ chosen by cross-validation on the training photographs. A readout from layer4 has up to 8,192 weights (2,048 maps × 2 × 2), fitted on 5,000 photographs, so the ridge penalty is what keeps it from memorizing noise

- The assignment's last part does this for the last block of the standard and the robust ResNet-50 - Computed for this lecture on the TVSD test photographs, once for every block; raw medians, not divided by each area's peak. The y axis starts at 0.35 - Each site: a ridge model from the block's features (each map average-pooled to 2 × 2, then flattened) to the response, λ chosen by cross-validation on the training photographs; scored by the Pearson correlation on the 100 test photographs, divided by the square root of the site's split-half reliability (Spearman–Brown corrected). Squared, it is the fraction of the site's repeatable variance that the readout predicts - In monkey N the V4 and IT peaks are one block apart and their intervals overlap: read the shapes, not the peak block - Bands: 95% intervals over 1,000 bootstrap draws of the test photographs. Peaks with intervals: V4 F 0.680 [0.611, 0.729], N 0.706 [0.645, 0.750]; IT F 0.759 [0.721, 0.790], N 0.812 [0.735, 0.852]; V1 F 0.638 at layer1.0, N 0.662 at layer3.2 - V1 is predicted almost equally by every block up to the last group, so its curve does not say where V1 sits: each map is pooled to 2 × 2, while a V1 site sees a small patch. Fits that keep each map's positions place V1 at the early blocks (Cadena et al. 2019) - Training matters: on the first lecture's IT sites, a ResNet-50 with random weights reaches about 0.30 against 0.52 trained (baselines lecture) - The last blocks fall because they serve the 1,000 ImageNet labels, the reason the assignment fine-tunes layer4 instead of reading it out as it is

- Your question from the first minute paper: is AI getting closer to the brain, or further away? Two figures, then a discussion

- Neural alignment: a linear fit from a network's features to monkey IT responses (recordings from Arcaro et al. 2020), scored on held-out images, the same kind of score as the block-by-area curve. ImageNet multi-label accuracy credits every valid label of an image; the gray band is the range of human accuracy - Red: CNNs trained on ImageNet-1K; yellow: adversarially trained CNNs; green: convolutional transformers; blue: vision transformers, with lighter shades for internet-scale training and gray for vision-and-language models. Purple stars are models added in 2026 by Akash Nagaraj - Once accuracy passes about 0.9, more accurate models predict IT less well, and the trend holds within families. Scale alone is not bringing networks closer to the brain - Yellow, the adversarially trained CNNs, are robust networks like the one the assignment compares with the standard ResNet-50 as models of IT

- A draft figure from my Annual Review of Vision Science chapter: the cohort of Linsley, Feng and Serre (2026), 363 released ImageNet classifiers (supervised CNNs and transformers, self-supervised, image–text and adversarially trained variants), scored with the Brain-Score protocols against four targets - Targets: (a) spatial response maps of face-selective IT neurons (Arcaro et al. 2020); (b) Triple-N (Li et al. 2026), Neuropixels recordings through IT including the category-selective patches, scene photographs; (c) TVSD (Papale et al. 2025), the assignment's data; (d) human ClickMe feature maps (Linsley et al. 2019). ρ is the rank correlation between accuracy and score, over all 363 and over the 84 above 92.7% accuracy - Scores are not divided by their ceilings here (the dashed line is each dataset's noise ceiling, or human-to-human agreement in d). Architecture and training vary together across the cohort, so this says whether accuracy gains bring alignment gains; it does not isolate what causes them

- The lab is running these experiments on TVSD and Triple-N now; the chapter's "future issues" are the same questions at full scale: how the structure of visual experience shapes what is learned, how resource constraints shape computation, which algorithms networks acquire, how to tell apart models that predict the same responses, and whether such relations hold across tasks and species - Controversial stimuli (Golan, Raju & Kriegeskorte 2020): images synthesized so that two models disagree about the category; a person's or a neuron's answer then favors one model. Prince et al. (2026) found that models with similar predictivity differ in how well their images control neurons - Scope for a team: one dataset, one manipulation, one measurement, controls named before the run; the write-up says what would have falsified the hypothesis - The project brief on Canvas lists the datasets, the compute, and the deadlines (teams by 11/19, proposal 11/24)

- The lecture can stop here; the slides after the minute paper are optional reading - Next lecture: learning without labels, the last lecture of Theme 2

- Schrimpf, Kubilius, Lee, Ratan Murty, Ajemian & DiCarlo (2020), Integrative benchmarking to advance neurally mechanistic models of human intelligence - Bashivan, Kar & DiCarlo (2019), Neural population control via deep image synthesis - Conwell, Prince, Kay, Alvarez & Konkle (2024), A large-scale examination of inductive biases shaping high-level visual representation in brains and machines - Goh, Cammarata, Voss, Carter, Petrov, Schubert, Radford & Olah (2021), Multimodal neurons in artificial neural networks - Geirhos, Jacobsen, Michaelis, Zemel, Brendel, Bethge & Wichmann (2020), Shortcut learning in deep neural networks - Templeton et al. (2024), Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet - Muttenthaler, Greff, Born, Spitzer, Kornblith, Mozer, Müller, Unterthiner & Lampinen (2025), Aligning machine and human visual representations across abstraction levels - Fel, Rodriguez, Linsley & Serre (2022), Harmonizing the object recognition strategies of deep neural networks with humans - Papale, Wang, Self et al. (2025), An extensive dataset of spiking activity to reveal the syntax of the ventral stream - Also in the optional slides at the end, and worth reading: Kamitani & Tong (2005) on decoding orientation, Horikawa et al. (2013) on decoding dreams - Before posting, skim the existing entries on your paper; your post must add something not already said

- Submitted on Canvas (Minute papers → Minute Paper 14); credit for a thoughtful attempt - Write the paper after the lecture, about what was covered in class; a paper about slides not reached, or written during the lecture, loses credit

- MVPA: multivoxel pattern analysis, the fMRI name for decoding - Linear predictivity ran from network to neuron. Readouts also run the other way: what does an area's activity say about the stimulus? The geometry lecture read animacy from IT neurons this way - Each polar plot: the grating shown at its top left; the outline, over 22 test samples per orientation, how often the readout chose each of the 8 orientations - A voxel is one fMRI measurement, a small cube of cortex a few millimeters across - Orientation columns: V1 neurons preferring one orientation sit in patches about half a millimeter wide, smaller than a voxel - Each voxel averages many columns with a slight imbalance; the readout combines these small biases. That was the original explanation; later work found much of the signal in coarse maps across V1, such as a preference for orientations pointing at the fixation point (Freeman et al. 2011), and the question is still debated - The figure's panels are from participant S2, 400 voxels of V1 and V2 - Trained and tested on separate runs, as in the baselines lecture - Decoding accuracy says the information is present and linearly readable, not that the brain uses it - Haxby et al. (2001) founded the method: patterns across ventral temporal cortex distinguish object categories, even outside the regions that respond most to each category

- Horikawa, Tamaki, Miyawaki & Kamitani (2013), Neural decoding of visual imagery during sleep - Participants were woken repeatedly in early sleep and asked what they had seen; decoders were trained on their awake responses to photographs - Suggests dream content uses the same visual representations as seeing