Learning without labels

Reader-friendly version of the 62 lecture slides: the lecture content in normal flow, with figure descriptions and data tables where the figure carries data. Open the presentation.

Slide 2

Learning without labels

How can a network learn useful features without category labels? Reconstruct the input, generate new samples, match views, predict what is hidden. Then: from images to video.

Return to contents

Slide 3

From AlexNet to today

Scatter plot of ImageNet multilabel accuracy, 60 to 97.5 percent, against training computation in floating-point operations from 10 to the 18 to 10 to the 24, for 40 image encoders released between 2012 and 2025. Accuracy rises from AlexNet near 62 percent to a cluster inside the gray band of trained human annotators, about 92 to 97 percent. Points are colored by objective: supervised black, self-supervised blue, image-text orange, video yellow; circles are CNNs, diamonds transformers, a square the MambaVision hybrid; gray markers are the 2012 to 2018 models. DINOv3, self-supervised, and several image-text transformers sit at the top right, inside the human band. Qwen3-VL is plotted separately at right because its training computation is unpublished.

Scatter plot of ImageNet multilabel accuracy, 60 to 97.5 percent, against training computation in floating-point operations from 10 to the 18 to 10 to the 24, for 40 image encoders released between 2012 and 2025. Accuracy rises from AlexNet near 62 percent to a cluster inside the gray band of trained human annotators, about 92 to 97 percent. Points are colored by objective: supervised black, self-supervised blue, image-text orange, video yellow; circles are CNNs, diamonds transformers, a square the MambaVision hybrid; gray markers are the 2012 to 2018 models. DINOv3, self-supervised, and several image-text transformers sit at the top right, inside the human band. Qwen3-VL is plotted separately at right because its training computation is unpublished.

Open full-size figure
Serre Annu Rev Vis Sci (in submission), Fig. 1 · Shankar et al. ICML 2020 · Siméoni et al. arXiv 2025 · Krizhevsky et al. NeurIPS 2012

Return to contents

Slide 4

What made scale pay off

Scatter plot of ImageNet multilabel accuracy, 60 to 97.5 percent, against training computation in floating-point operations from 10 to the 18 to 10 to the 24, for 40 image encoders released between 2012 and 2025. Accuracy rises from AlexNet near 62 percent to a cluster inside the gray band of trained human annotators, about 92 to 97 percent. Points are colored by objective: supervised black, self-supervised blue, image-text orange, video yellow; circles are CNNs, diamonds transformers, a square the MambaVision hybrid; gray markers are the 2012 to 2018 models. DINOv3, self-supervised, and several image-text transformers sit at the top right, inside the human band. Qwen3-VL is plotted separately at right because its training computation is unpublished.

Scatter plot of ImageNet multilabel accuracy, 60 to 97.5 percent, against training computation in floating-point operations from 10 to the 18 to 10 to the 24, for 40 image encoders released between 2012 and 2025. Accuracy rises from AlexNet near 62 percent to a cluster inside the gray band of trained human annotators, about 92 to 97 percent. Points are colored by objective: supervised black, self-supervised blue, image-text orange, video yellow; circles are CNNs, diamonds transformers, a square the MambaVision hybrid; gray markers are the 2012 to 2018 models. DINOv3, self-supervised, and several image-text transformers sit at the top right, inside the human band. Qwen3-VL is plotted separately at right because its training computation is unpublished.

Open full-size figure
Serre Annu Rev Vis Sci (in submission), Fig. 1 · Shankar et al. ICML 2020 · Siméoni et al. arXiv 2025

Return to contents

Slide 5

The label problem

Return to contents

Slide 6

What does a network learn from its inputs alone?

Train a network to copy its input through a narrow layer, a bottleneck. The input must be compressed: what does the network keep?

Return to contents

Slide 7

The autoencoder

A 1024–32–1024 autoencoder on a held-out CelebA face, drawn unit by unit: a column of input units standing for the 1024 pixels, magenta weight lines W1 fanning into 32 green code units shaded by activity, weight lines W2 fanning out to output units, and the blurrier reconstructed face on the right

A 1024–32–1024 autoencoder on a held-out CelebA face, drawn unit by unit: a column of input units standing for the 1024 pixels, magenta weight lines W1 fanning into 32 green code units shaded by activity, weight lines W2 fanning out to output units, and the blurrier reconstructed face on the right

Open full-size figure

Computed: a 1024–32–1024 network trained on 60,000 CelebA faces, 32 × 32, grayscale; a held-out face

Computed on CelebA

Return to contents

Slide 8

The reconstruction loss

A 1024–32–1024 autoencoder on a held-out CelebA face, drawn unit by unit: a column of input units standing for the 1024 pixels, magenta weight lines W1 fanning into 32 green code units shaded by activity, weight lines W2 fanning out to output units, and the blurrier reconstructed face on the right

A 1024–32–1024 autoencoder on a held-out CelebA face, drawn unit by unit: a column of input units standing for the 1024 pixels, magenta weight lines W1 fanning into 32 green code units shaded by activity, weight lines W2 fanning out to output units, and the blurrier reconstructed face on the right

Open full-size figure

ℓ=12 ∥𝐱^−𝐱∥2,L=1N∑i12 ∥𝐱^(i)−𝐱(i)∥2\ell = \tfrac12\,\|\hat{\mathbf{x}} - \mathbf{x}\|^2, \qquad \mathcal{L} = \frac{1}{N}\sum_{i}\tfrac12\,\|\hat{\mathbf{x}}^{(i)} - \mathbf{x}^{(i)}\|^2

Computed on CelebA

Return to contents

Slide 9

What the encoder does: a projection

The same held-out face, input units, weight fan and 32-unit code as the last slide. To the right, four rows of the trained encoder weights W1 drawn as 32 by 32 images showing face structure, in a two by two grid, each joined by a line to its code unit and labeled with that unit's value. Further right, a sketch: a cloud of points, a tilted plane, and thin lines dropping each point onto the plane

The same held-out face, input units, weight fan and 32-unit code as the last slide. To the right, four rows of the trained encoder weights W1 drawn as 32 by 32 images showing face structure, in a two by two grid, each joined by a line to its code unit and labeled with that unit's value. Further right, a sketch: a cloud of points, a tilted plane, and thin lines dropping each point onto the plane

Open full-size figure
Computed on CelebA

Return to contents

Slide 10

With no gg, the autoencoder recovers PCA's subspace

Let 𝐕k\mathbf{V}_k hold the top kk eigenvectors of the data's covariance, as columns, with 𝐱\mathbf{x} centered:

With no gg, the autoencoder recovers PCA's subspace — table
Code Reconstruction
PCA 𝐡=𝐕k⊤𝐱\mathbf{h} = \mathbf{V}_k^\top\mathbf{x} 𝐱^=𝐕k𝐕k⊤𝐱\hat{\mathbf{x}} = \mathbf{V}_k\mathbf{V}_k^\top\mathbf{x}
Linear autoencoder 𝐡=𝐖1𝐱\mathbf{h} = \mathbf{W}_1\mathbf{x} 𝐱^=𝐖2𝐖1𝐱\hat{\mathbf{x}} = \mathbf{W}_2\mathbf{W}_1\mathbf{x}

Return to contents

Slide 11

Same subspace, different rows

Left, a held-out face and its reconstructions from 32 numbers by PCA and by a linear autoencoder: the two look the same, both R squared 0.809, and the largest angle between their subspaces is below 0.001 degrees. Right, four PCA eigenvectors above four rows of W1, each drawn as a 32 by 32 image: the rows differ from the eigenvectors

Left, a held-out face and its reconstructions from 32 numbers by PCA and by a linear autoencoder: the two look the same, both R squared 0.809, and the largest angle between their subspaces is below 0.001 degrees. Right, four PCA eigenvectors above four rows of W1, each drawn as a 32 by 32 image: the rows differ from the eigenvectors

Open full-size figure
Computed on CelebA

Return to contents

Slide 12

Nonlinear autoencoders go beyond PCA

Two panels of 300 points on a noisy half circle in two dimensions, each point joined by a faint line to its reconstruction. Both networks have the same layers, 2-64-64-1-64-64-2. Left, no nonlinearity: the reconstructions lie on a straight line, mean squared error 0.107. Right, tanh: the reconstructions follow the arc, mean squared error 0.002

Two panels of 300 points on a noisy half circle in two dimensions, each point joined by a faint line to its reconstruction. Both networks have the same layers, 2-64-64-1-64-64-2. Left, no nonlinearity: the reconstructions lie on a straight line, mean squared error 0.107. Right, tanh: the reconstructions follow the arc, mean squared error 0.002

Open full-size figure

Computed: toy data, 300 points on a noisy half circle

Return to contents

Slide 13

Try it: compress a face

Return to contents

Slide 14

Sparse coding: reconstruct from a few basis functions

The same reconstruction loss, plus L1 regularization on the code, of strength λ\lambda. Unlike L2, L1 sets small values to exactly zero, so no bottleneck is needed, even with more functions than pixels (overcomplete):

ℓ=12 ∥𝐱^−𝐱∥2+λ ∥𝐡∥1\ell = \tfrac12\,\|\hat{\mathbf{x}} - \mathbf{x}\|^2 + \lambda\,\|\mathbf{h}\|_1

One 16 by 16 whitened image patch, shown to be approximately equal to the sum of three learned basis functions, each an oriented stripe pattern multiplied by its coefficient, plus a few more basis functions. To the right, a stem plot of the patch's 256 code coefficients, almost all exactly zero

One 16 by 16 whitened image patch, shown to be approximately equal to the sum of three learned basis functions, each an oriented stripe pattern multiplied by its coefficient, plus a few more basis functions. To the right, a stem plot of the patch's 256 code coefficients, almost all exactly zero

Open full-size figure

Return to contents

Slide 15

L1 against L2 on the same patch

Two stem plots of the code of one patch, each on its own axis. Left, the L2 code: all 256 coefficients are not zero, the largest about 2. Right, the L1 code, with the same reconstruction quality: only 9 of 256 are not zero, the largest about 8

Two stem plots of the code of one patch, each on its own axis. Left, the L2 code: all 256 coefficients are not zero, the largest about 2. Right, the L1 code, with the same reconstruction quality: only 9 of 256 are not zero, the largest about 8

Open full-size figure

Return to contents

Slide 16

From reconstructing to generating

An autoencoder reconstructs the image it is given. Can it make a new one?

Return to contents

Slide 17

What if we keep only the decoder?

Two rows of eight square faces from the decoder of a plain autoencoder. Top row, decoded from the codes of real held-out faces: blurred but natural faces. Bottom row, decoded from random codes: faces with severe distortion, grainy and with harsh black and white areas

Two rows of eight square faces from the decoder of a plain autoencoder. Top row, decoded from the codes of real held-out faces: blurred but natural faces. Bottom row, decoded from random codes: faces with severe distortion, grainy and with harsh black and white areas

Open full-size figure

Computed: same faces, a larger autoencoder, 1024–512–32–512–1024

Computed on CelebA

Return to contents

Slide 18

Variational autoencoder: sampling new images

Half of a variational autoencoder drawn unit by unit. A held-out face x feeds a column of input units standing for 1024 pixels, then weights to 512 hidden ReLU units, then two columns of 32 circles, the means and the spreads for this face. Those feed a box where a random epsilon makes one draw z, the 32-unit green code. To the right, a small decoder trapezoid and the reconstructed face

Half of a variational autoencoder drawn unit by unit. A held-out face x feeds a column of input units standing for 1024 pixels, then weights to 512 hidden ReLU units, then two columns of 32 circles, the means and the spreads for this face. Those feed a box where a random epsilon makes one draw z, the 32-unit green code. To the right, a small decoder trapezoid and the reconstructed face

Open full-size figure

Return to contents

Slide 19

What the VAE gives you: new faces

Three rows of square faces. First row, a plain autoencoder decoding random codes: recognizable but distorted, grainy, overly dark and light faces. Second row, a variational autoencoder decoding the same codes: more coherent, blurry faces. Third row, a walk between two held-out faces in the code of the variational autoencoder: eight faces changing gradually

Three rows of square faces. First row, a plain autoencoder decoding random codes: recognizable but distorted, grainy, overly dark and light faces. Second row, a variational autoencoder decoding the same codes: more coherent, blurry faces. Third row, a walk between two held-out faces in the code of the variational autoencoder: eight faces changing gradually

Open full-size figure
Computed on CelebA

Return to contents

Slide 20

Try it: walk the code

Return to contents

Slide 21

Beyond the VAE

Return to contents

Slide 22

From reconstruction to prediction

Self-supervised learning: first match views of an image (SimCLR, DINO), then predict what is hidden (MAE). The goal: approach the accuracy of a fully supervised network, without its labels

Return to contents

Slide 23

Equal reconstruction, unequal features

From Balestriero and LeCun: two ResNet-34 autoencoders give near-identical reconstructions of stereos and equal reconstruction loss, yet test accuracy with a linear readout is about 60 against 87 percent

From Balestriero and LeCun: two ResNet-34 autoencoders give near-identical reconstructions of stereos and equal reconstruction loss, yet test accuracy with a linear readout is about 60 against 87 percent

Open full-size figure
  • Good reconstructions do not imply good features: two autoencoders with the same reconstruction loss score about 60% and 87% on a linear readout (left; the better one also got a supervised signal)
  • So copying the input does not guarantee features useful for other tasks. What else could the target be, with no labels?

Return to contents

Slide 24

Self-supervised learning

Return to contents

Slide 25

Early pretext tasks

One cat photograph transformed for four tasks. Rotation: the photo turned 90 degrees, predict the angle. Jigsaw: its nine tiles shuffled, predict the shuffle. Relative position: two neighboring patches, predict where the red one lies relative to the blue. Colorization: the gray-level photo, predict the colors

One cat photograph transformed for four tasks. Rotation: the photo turned 90 degrees, predict the angle. Jigsaw: its nine tiles shuffled, predict the shuffle. Relative position: two neighboring patches, predict where the red one lies relative to the blue. Colorization: the gray-level photo, predict the colors

Open full-size figure

Return to contents

Slide 26

Three classes of methods

Schematic from the Annual Review chapter: two views of a cat image, with representations pulled together and other images pushed apart

Schematic from the Annual Review chapter: two views of a cat image, with representations pulled together and other images pushed apart

Open full-size figure

Contrastive (SimCLR): match two views, push others apart

Schematic from the Annual Review chapter: one view of a cat through an encoder predicts the output of a running-average copy on another view

Schematic from the Annual Review chapter: one view of a cat through an encoder predicts the output of a running-average copy on another view

Open full-size figure

Self-distillation (DINO): match a slowly updated copy of itself

Schematic from the Annual Review chapter: a cat image with most patches hidden goes through an encoder and decoder and is compared with the original

Schematic from the Annual Review chapter: a cat image with most patches hidden goes through an encoder and decoder and is compared with the original

Open full-size figure

Masked prediction (MAE): fill in hidden patches

Serre Annu Rev Vis Sci (in submission), Fig. 2c–e · Chen et al. ICML 2020 · Caron et al. ICCV 2021 · He et al. CVPR 2022

Return to contents

Slide 27

Attraction: two views of one image should agree

One cat photograph and five random views of it: a crop, a crop and flip, a color-jittered version, a gray-level version and a blurred version

One cat photograph and five random views of it: a crop, a crop and flip, a color-jittered version, a gray-level version and a blurred version

Open full-size figure

Return to contents

Slide 28

Add repulsion: contrastive learning (SimCLR)

One crop of a cat on the left, seven candidates on the right: a koala, a dog, a car, another crop of the same cat outlined in red, bananas, a chair and a teapot. Bars show a schematic probability for each, 0.86 for the other cat crop and at most 0.06 for the others

One crop of a cat on the left, seven candidates on the right: a koala, a dog, a car, another crop of the same cat outlined in red, bananas, a chair and a teapot. Bars show a schematic probability for each, 0.86 for the other cat crop and at most 0.06 for the others

Open full-size figure

Return to contents

Slide 29

SimCLR: where it excels

Horizontal bar chart of ImageNet accuracy of a linear readout when training uses one kind of augmentation: rotate 2.6 percent, blur 2.6, color 18.8, crop 33.1, and crop plus color together 56.3

Horizontal bar chart of ImageNet accuracy of a linear readout when training uses one kind of augmentation: rotate 2.6 percent, blur 2.6, color 18.8, crop 33.1, and crop plus color together 56.3

Open full-size figure
  • Linear readout on a frozen ResNet-50: 69% on ImageNet, against 60 to 64% for earlier methods
  • Fine-tuning with few labels: pretrain with no labels, then fine-tune on 1% of the labels. Top-5 accuracy 75%, against 48% for supervised training on that same 1%
  • It depends on the views (figure). No single augmentation is enough. Crop plus color: 56%
Chen et al. ICML 2020 · Fig. 5, replotted

Return to contents

Slide 30

SimCLR: where it fails

Schematic from Chuang et al.: a golden retriever, an augmented view of it as the positive, and negatives: a cat, an airplane and a second golden retriever outlined in red as a false negative

Schematic from Chuang et al.: a golden retriever, an augmented view of it as the positive, and negatives: a cat, an airplane and a second golden retriever outlined in red as a false negative

Open full-size figure
  • False negatives: negatives are sampled at random, with no labels, so two images of one class, such as two dogs, get pushed apart. With 30 negatives on CIFAR-10: about 80% accuracy; 88% when sampled from different classes
  • Batch size: the batch supplies the negatives, and a larger batch gives more of them, including harder ones. Default 4096 images; smaller batches can come close

Return to contents

Slide 31

Second class: self-distillation

DINO: match a slowly updated copy of the network, with no repulsion

Return to contents

Slide 32

DINO: drop repulsion, match a slowly updated copy

Schematic from the Annual Review chapter: one view of a cat goes through the encoder, another view goes through a dashed box labelled encoder, running-average copy. The first output is trained to predict the second, the target, compared in features

Schematic from the Annual Review chapter: one view of a cat goes through the encoder, another view goes through a dashed box labelled encoder, running-average copy. The first output is trained to predict the second, the target, compared in features

Open full-size figure
Caron et al. ICCV 2021 · Serre Annu Rev Vis Sci (in submission), Fig. 2e · Grill et al. NeurIPS 2020

Return to contents

Slide 33

What a ResNet learns with no labels

Springer spaniel, church, gas pump, tench and parachute photographs, each with first-PC and center-seed cosine maps from a DINO and a supervised ResNet-50. The maps are patchy and do not reliably cover the object

Springer spaniel, church, gas pump, tench and parachute photographs, each with first-PC and center-seed cosine maps from a DINO and a supervised ResNet-50. The maps are patchy and do not reliably cover the object

Open full-size figure
  • The last layer of a DINO ResNet-50 is a 25 × 25 map, one feature vector per location
  • First PC: PCA over one image's locations: its largest contrast in features
  • Cosine to a seed: which locations have feature vectors pointing the same way as at one point
  • With no labels, the maps look much like those of a ResNet-50 trained on 1.3 million labeled images. In both, similar-looking locations group together, without reliably covering the whole object
Computed on five ImageNet photographs · Caron et al. ICCV 2021 · He et al. CVPR 2016 · Geirhos et al. ICLR 2019

Return to contents

Slide 34

The same objective on a transformer

Figure 1 of Caron et al.: photographs with the self-attention map of a DINO vision transformer beneath each, outlining the object with no label, mask or box given

Figure 1 of Caron et al.: photographs with the self-attention map of a DINO vision transformer beneath each, outlining the object with no label, mask or box given

Open full-size figure

Return to contents

Slide 35

DINOv2: the same parts, matched across images

Figure 1 of Oquab et al.: four columns of photographs with PCA maps of patch features beside them, colored by the first three components. Eagles and airplanes, elephants and a statue, horses and a drawing, vehicles: corresponding parts share colors

Figure 1 of Oquab et al.: four columns of photographs with PCA maps of patch features beside them, colored by the first three components. Eagles and airplanes, elephants and a statue, horses and a drawing, vehicles: corresponding parts share colors

Open full-size figure

Return to contents

Slide 36

DINOv3: sharp features at high resolution

Left, Figure 3 of Simeoni et al.: a market stall of fruit surrounded by cosine similarity maps, each lighting up the patches like one marked patch. Right, Figure 1c and d: PCA maps of a cat and of aerial images, showing fine detail

Left, Figure 3 of Simeoni et al.: a market stall of fruit surrounded by cosine similarity maps, each lighting up the patches like one marked patch. Right, Figure 1c and d: PCA maps of a cat and of aerial images, showing fine detail

Open full-size figure
Siméoni et al. arXiv 2025 · Figs. 3, 1c–d

Return to contents

Slide 37

DINO to DINOv3: mostly the same recipe, at scale

DINO to DINOv3: mostly the same recipe, at scale — table
DINO (2021) DINOv2 (2023) DINOv3 (2025)
Images 1.3 million 142 million 1.7 billion
Parameters 85 million 1.1 billion 6.7 billion
Added slow copy, centering, sharpening masked patches Gram anchoring
ImageNet readout 80% 87% 88%

Return to contents

Slide 38

Third class: masked prediction

DINOv2 predicts hidden features; MAE reconstructs hidden pixels. No views to match

Return to contents

Slide 39

Masked autoencoder (MAE): drop the views

Schematic from the Annual Review chapter: a cat image with most patches hidden goes through an encoder and a decoder; the output is compared in pixels with the original

Schematic from the Annual Review chapter: a cat image with most patches hidden goes through an encoder and a decoder; the output is compared in pixels with the original

Open full-size figure
He et al. CVPR 2022 · Serre Annu Rev Vis Sci (in submission), Fig. 2d

Return to contents

Slide 40

MAE: real reconstructions

Bus, vase, zebras, mushrooms and vegetables reconstructed from 75, 85 and 95 percent hidden patches. Reconstructions blur as masking rises, and at 95 percent the green pepper disappears

Bus, vase, zebras, mushrooms and vegetables reconstructed from 75, 85 and 95 percent hidden patches. Reconstructions blur as masking rises, and at 95 percent the green pepper disappears

Open full-size figure
  • From 25% of the patches, a masked autoencoder (a transformer) reconstructs the bus and the zebras
  • At 95% hidden, it still draws a plausible bus. But the green pepper is gone: filled in from what is typical, not from what was there

Return to contents

Slide 41

SimCLR, DINO, MAE: each removes a dependence

SimCLR, DINO, MAE: each removes a dependence — table
Attraction + repulsion
(SimCLR)
Attraction only
(DINO)
Masked prediction
(MAE)
Removes labels negatives views
Prevents collapse by repulsion slow copy, centering, sharpening targets differ image to image
Excels few labels frozen features: 78% against 68% fine-tuning: 83.6% against 82.8%
Fails false negatives; needs the right views delicate balance frozen readout

On the same transformer (ViT-B/16), with each paper's recipe, DINO gives stronger frozen features; MAE gains more from fine-tuning

Return to contents

Slide 42

One recipe: pretrain once, read out cheaply

Figure 1a of Simeoni et al. 2025: ImageNet accuracy of a linear readout by year, 2012 to 2025. Supervised models rise from about 60 to near 90 percent. Self-supervised models start near 38 percent around 2018 and rise steeply, reaching the same level by 2025

Figure 1a of Simeoni et al. 2025: ImageNet accuracy of a linear readout by year, 2012 to 2025. Supervised models rise from about 60 to near 90 percent. Self-supervised models start near 38 percent around 2018 and rise steeply, reaching the same level by 2025

Open full-size figure
  • Pretrain once with no labels, then fit a linear readout on the frozen encoder
  • Figure: that readout's ImageNet accuracy by year. SL: supervised. WSL: weakly supervised, from hashtags or captions. SSL: self-supervised
  • Self-supervised pretraining now approaches supervised models on ImageNet: labels are needed only for the readout
  • Foundation models work this way: large models pretrained on broad data, typically without labels, then adapted to many tasks

Return to contents

Slide 43

Back to the scaling figure

Scatter plot of ImageNet multilabel accuracy, 60 to 97.5 percent, against training computation in floating-point operations from 10 to the 18 to 10 to the 24, for 40 image encoders released between 2012 and 2025. Accuracy rises from AlexNet near 62 percent to a cluster inside the gray band of trained human annotators, about 92 to 97 percent. Points are colored by objective: supervised black, self-supervised blue, image-text orange, video yellow; circles are CNNs, diamonds transformers, a square the MambaVision hybrid; gray markers are the 2012 to 2018 models. DINOv3, self-supervised, and several image-text transformers sit at the top right, inside the human band. Qwen3-VL is plotted separately at right because its training computation is unpublished.

Scatter plot of ImageNet multilabel accuracy, 60 to 97.5 percent, against training computation in floating-point operations from 10 to the 18 to 10 to the 24, for 40 image encoders released between 2012 and 2025. Accuracy rises from AlexNet near 62 percent to a cluster inside the gray band of trained human annotators, about 92 to 97 percent. Points are colored by objective: supervised black, self-supervised blue, image-text orange, video yellow; circles are CNNs, diamonds transformers, a square the MambaVision hybrid; gray markers are the 2012 to 2018 models. DINOv3, self-supervised, and several image-text transformers sit at the top right, inside the human band. Qwen3-VL is plotted separately at right because its training computation is unpublished.

Open full-size figure
Serre Annu Rev Vis Sci (in submission) · Siméoni et al. arXiv 2025 · Conwell et al. Nat Commun 2024

Return to contents

Slide 44

Does the brain need labels?

We learned useful features with no labels, from images and video. How do these models compare with biological vision?

Return to contents

Slide 45

Which of these methods is brain-like?

Serre Annu Rev Vis Sci (in submission)

Return to contents

Slide 46

More accurate is not reliably more brain-like

Five scatter plots in a row of ImageNet multilabel accuracy against alignment with biological vision for 363 classifiers. IT prediction on three datasets: correlation 0.37 for Arcaro, 0.66 for TVSD, minus 0.08 for Triple-N, all well below each measure's ceiling. Human feature use: minus 0.77. Human similarity judgments: minus 0.49. DINOv3 and Qwen3-VL are marked

Five scatter plots in a row of ImageNet multilabel accuracy against alignment with biological vision for 363 classifiers. IT prediction on three datasets: correlation 0.37 for Arcaro, 0.66 for TVSD, minus 0.08 for Triple-N, all well below each measure's ceiling. Human feature use: minus 0.77. Human similarity judgments: minus 0.49. DINOv3 and Qwen3-VL are marked

Open full-size figure
Serre Annu Rev Vis Sci (in submission), Fig. 4

Return to contents

Slide 47

From images to video

Toward more ecologically valid data: a continuous stream of experience

Return to contents

Slide 48

From static images to a stream of experience

Five pairs of images of one cat, input on the left and target on the right. Reconstruct the input: the photo and itself. Predict what is hidden: a masked photo and the original. Match two views of one photo: two different crops. These three are bracketed as static images, what most models learn from. Match views close in time and predict the next view: slightly shifted frames, bracketed as a stream, what animals learn from

Five pairs of images of one cat, input on the left and target on the right. Reconstruct the input: the photo and itself. Predict what is hidden: a masked photo and the original. Match two views of one photo: two different crops. These three are bracketed as static images, what most models learn from. Match views close in time and predict the next view: slightly shifted frames, bracketed as a stream, what animals learn from

Open full-size figure
Serre Annu Rev Vis Sci (in submission) · Photographs: THINGSplus

Return to contents

Slide 49

Learning from time: one child's head camera

Head-camera video frames stacked along a time arrow. A group showing a white cat on a green cushion is labelled Class A, and a later group showing a living room with a couch is labelled Class B. The depth of each stack is the segment length

Head-camera video frames stacked along a time arrow. A group showing a white cat on a green cushion is labelled Class A, and a later group showing a living room with a couch is labelled Class B. The depth of each stack is the segment length

Open full-size figure
  • The data: about 200 hours of video from a camera on one child's head, between 6 and 32 months. No labels
  • The task: cut the video into segments about 5 minutes long and train a convolutional network (MobileNetV2) to say which segment a frame came from
  • Time alone supplies the training labels. It works on that child's own objects: better than a network trained with ImageNet labels. On new toys, the ImageNet network wins

Return to contents

Slide 50

Object constancy depends on temporal experience

Change how views follow each other in time, and tolerance changes:

These are among the few neural laws of visual learning established so far. A model of temporal learning must reproduce them

Return to contents

Slide 51

Where matching views fails, and what prediction adds

Scaling has so far increased the amount of experience models receive, not its kind. A project question

Garimella et al. ICLR 2024 · Ashok et al. arXiv 2026 · Serre Annu Rev Vis Sci (in submission)

Return to contents

Slide 52

Recap

Return to contents

Slide 53

Looking ahead

Return to contents

Slide 54

Further reading for the research trail

Optional. Any paper from any lecture qualifies; these are preferred.

Return to contents

Slide 55

Minute paper

Submit on Canvas → Minute papers → Minute Paper 15 (access code read out in class).

Write three brief points in your own words:

  1. Pick one method from today (autoencoder, SimCLR, DINO or masked prediction). In your own words: what is the network trained to predict, and where does that target come from, if not from a human label?
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents

Slide 56

Going further

The VAE loss and the contrastive loss in equations and code. More on visual experience and video

Return to contents

Slide 57

The VAE loss

ℓ=12σx2 ∥𝐱^−𝐱∥2⏟reconstruction  +  β  12∑j=1k(μj2+σj2−log⁡σj2−1)⏟KL divergence to a standard Gaussian\ell = \underbrace{\tfrac{1}{2\sigma_x^2}\,\|\hat{\mathbf{x}} - \mathbf{x}\|^2}_{\text{reconstruction}} \;+\; \beta\;\underbrace{\tfrac12\sum_{j=1}^{k}\big(\mu_j^2 + \sigma_j^2 - \log\sigma_j^2 - 1\big)}_{\text{KL divergence to a standard Gaussian}}

Return to contents

Slide 58

The InfoNCE loss

Here 𝐳(i)\mathbf{z}^{(i)} is the head's output for view ii, not a VAE sample. Cosine similarity s(𝐮,𝐯)=𝐮⊤𝐯∥𝐮∥ ∥𝐯∥s(\mathbf{u},\mathbf{v}) = \dfrac{\mathbf{u}^\top\mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|} and temperature τ\tau. For a positive pair (i,j)(i,j):

ℓ(i,j)=−log⁡exp⁡ ⁣(s(𝐳(i),𝐳(j))/τ)∑r≠iexp⁡ ⁣(s(𝐳(i),𝐳(r))/τ)\ell^{(i,j)} = -\log\frac{\exp\!\big(s(\mathbf{z}^{(i)},\mathbf{z}^{(j)})/\tau\big)}{\sum_{r\neq i}\exp\!\big(s(\mathbf{z}^{(i)},\mathbf{z}^{(r)})/\tau\big)}

Return to contents

Slide 59

PyTorch: the InfoNCE loss

import torch import torch.nn.functional as F def info_nce_loss(z, tau=0.5): # z: (2B, d), two augmented views stacked; rows 0..B-1 pair with B..2B-1 z = F.normalize(z, dim=1) # cosine similarity = dot of unit vectors sim = z @ z.T / tau # (2B, 2B) similarity matrix n = z.shape[0] sim.fill_diagonal_(float("-inf")) # a view is never its own positive targets = (torch.arange(n) + n // 2) % n # the positive of row i is its other view return F.cross_entropy(sim, targets) # softmax cross-entropy

Return to contents

Slide 60

How much visual experience?

Return to contents

Slide 61

Three learning signals in a stream

Three learning signals in a stream — table
Signal What it is What it can teach
Temporal continuity an object persists across successive views object constancy: recognizing an object despite changes in its retinal image
Common fate parts that move together belong together which regions form one object
Predictability one view regularly follows another what causes change: moving objects and their interactions
Serre Annu Rev Vis Sci (in submission) · Sevilla-Lara et al. WACV 2021

Return to contents

Slide 62

Models that learn from video and generate it

Bardes et al. TMLR 2024 · Sevilla-Lara et al. WACV 2021 · Zhou et al. 2026 (our lab) · Serre Annu Rev Vis Sci (in submission)

Return to contents