Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 14 · Theme 2: Learning representations: from neurons to deep networks

Learning without labels

Thursday, November 5 · Fall 2026

Learning without labels

How can a network learn useful features without category labels? Reconstruct the input, generate new samples, match views, predict what is hidden. Then: from images to video.

From AlexNet to today

Scatter plot of ImageNet multilabel accuracy, 60 to 97.5 percent, against training computation in floating-point operations from 10 to the 18 to 10 to the 24, for 40 image encoders released between 2012 and 2025. Accuracy rises from AlexNet near 62 percent to a cluster inside the gray band of trained human annotators, about 92 to 97 percent. Points are colored by objective: supervised black, self-supervised blue, image-text orange, video yellow; circles are CNNs, diamonds transformers, a square the MambaVision hybrid; gray markers are the 2012 to 2018 models. DINOv3, self-supervised, and several image-text transformers sit at the top right, inside the human band. Qwen3-VL is plotted separately at right because its training computation is unpublished.

  • 40 image encoders, networks that turn an image into features, from AlexNet in 2012 to 2025, measured alike. Accuracy climbs from about 60% to the range of trained annotators (gray)
  • DINOv3 took about 300,000 times the computation of AlexNet. At AlexNet's 2012 rate, that is about 4,500 years of training
Serre Annu Rev Vis Sci (in submission), Fig. 1 · Shankar et al. ICML 2020 · Siméoni et al. arXiv 2025 · Krizhevsky et al. NeurIPS 2012

What made scale pay off

Scatter plot of ImageNet multilabel accuracy, 60 to 97.5 percent, against training computation in floating-point operations from 10 to the 18 to 10 to the 24, for 40 image encoders released between 2012 and 2025. Accuracy rises from AlexNet near 62 percent to a cluster inside the gray band of trained human annotators, about 92 to 97 percent. Points are colored by objective: supervised black, self-supervised blue, image-text orange, video yellow; circles are CNNs, diamonds transformers, a square the MambaVision hybrid; gray markers are the 2012 to 2018 models. DINOv3, self-supervised, and several image-text transformers sit at the top right, inside the human band. Qwen3-VL is plotted separately at right because its training computation is unpublished.

  • Self-supervised learning (blue) trains on huge image collections with no category labels; image–caption models (orange) skip them too. The most accurate models learned without category labels. Today
  • The transformer (diamonds): an architecture that trains well at scale. A later lecture
Serre Annu Rev Vis Sci (in submission), Fig. 1 · Shankar et al. ICML 2020 · Siméoni et al. arXiv 2025

The label problem

  • Most networks so far learned from a human label: each input came with its class label yy. ImageNet alone has 1.2 million images labeled by hand
  • An infant sees millions of scenes and is told the name of almost none. Much of what a visual system learns must come without labels
  • Unsupervised learning, first today: no labels. PCA, the autoencoder and sparse coding find structure in the inputs alone. Generative models: generate new samples that follow the same distribution as the training data
  • Self-supervised learning, second: make training targets from the data, hide part and predict it, or match two views. No human labels, and the machinery of supervised learning

What does a network learn from its inputs alone?

Train a network to copy its input through a narrow layer, a bottleneck. The input must be compressed: what does the network keep?

The autoencoder

A 1024–32–1024 autoencoder on a held-out CelebA face, drawn unit by unit: a column of input units standing for the 1024 pixels, magenta weight lines W1 fanning into 32 green code units shaded by activity, weight lines W2 fanning out to output units, and the blurrier reconstructed face on the right

  • Units and weights as usual. The encoder maps x\mathbf{x} (DD numbers) to a code h\mathbf{h} (kk numbers); the decoder reconstructs x^\hat{\mathbf{x}}. With k<Dk<D (undercomplete), the bottleneck forces compression: here, 1024 pixels to 32 numbers

Computed: a 1024–32–1024 network trained on 60,000 CelebA faces, 32 × 32, grayscale; a held-out face

Computed on CelebA

The reconstruction loss

A 1024–32–1024 autoencoder on a held-out CelebA face, drawn unit by unit: a column of input units standing for the 1024 pixels, magenta weight lines W1 fanning into 32 green code units shaded by activity, weight lines W2 fanning out to output units, and the blurrier reconstructed face on the right

ℓ=12 ∥x^−x∥2,L=1N∑i12 ∥x^(i)−x(i)∥2\ell = \tfrac12\,\|\hat{\mathbf{x}} - \mathbf{x}\|^2, \qquad \mathcal{L} = \frac{1}{N}\sum_{i}\tfrac12\,\|\hat{\mathbf{x}}^{(i)} - \mathbf{x}^{(i)}\|^2

  • The target is the input, with no labels: unsupervised learning. Train with backprop: the error goes back through the decoder, then the encoder
Computed on CelebA

What the encoder does: a projection

The same held-out face, input units, weight fan and 32-unit code as the last slide. To the right, four rows of the trained encoder weights W1 drawn as 32 by 32 images showing face structure, in a two by two grid, each joined by a line to its code unit and labeled with that unit's value. Further right, a sketch: a cloud of points, a tilted plane, and thin lines dropping each point onto the plane

  • Each face is a point in 1024 dimensions (sketched in 3). Each row of W1\mathbf{W}_1 is a direction; h\mathbf{h} gives coordinates along 32 of them, passed through gg: a projection from 1024 to 32
  • Remember PCA: the directions of maximal variance span the subspace with the least squared reconstruction error. The autoencoder also minimizes squared error, with gg and free weights. Is it PCA?
Computed on CelebA

With no gg, the autoencoder recovers PCA's subspace

Let Vk\mathbf{V}_k hold the top kk eigenvectors of the data's covariance, as columns, with x\mathbf{x} centered:

Code Reconstruction
PCA h=Vk⊤x\mathbf{h} = \mathbf{V}_k^\top\mathbf{x} x^=VkVk⊤x\hat{\mathbf{x}} = \mathbf{V}_k\mathbf{V}_k^\top\mathbf{x}
Linear autoencoder h=W1x\mathbf{h} = \mathbf{W}_1\mathbf{x} x^=W2W1x\hat{\mathbf{x}} = \mathbf{W}_2\mathbf{W}_1\mathbf{x}
  • PCA restricts itself to orthonormal directions; the linear autoencoder does not. Yet the best W2W1\mathbf{W}_2\mathbf{W}_1 for squared error is the same projection, VkVk⊤\mathbf{V}_k\mathbf{V}_k^\top. With no nonlinearity, the autoencoder recovers PCA's subspace
  • The product is fixed, not each matrix. For any invertible k×kk \times k matrix A\mathbf{A}: W1=AVk⊤\mathbf{W}_1 = \mathbf{A}\mathbf{V}_k^\top and W2=VkA−1\mathbf{W}_2 = \mathbf{V}_k\mathbf{A}^{-1} work equally well, since A−1A\mathbf{A}^{-1}\mathbf{A} cancels. With A=I\mathbf{A} = \mathbf{I}, the rows of W1\mathbf{W}_1 are the eigenvectors

Same subspace, different rows

Left, a held-out face and its reconstructions from 32 numbers by PCA and by a linear autoencoder: the two look the same, both R squared 0.809, and the largest angle between their subspaces is below 0.001 degrees. Right, four PCA eigenvectors above four rows of W1, each drawn as a 32 by 32 image: the rows differ from the eigenvectors

  • Same reconstruction, different weights: the rows of W1\mathbf{W}_1 are not the eigenvectors. Training with backprop from another random initialization gives other rows, and the same reconstruction
Computed on CelebA

Nonlinear autoencoders go beyond PCA

Two panels of 300 points on a noisy half circle in two dimensions, each point joined by a faint line to its reconstruction. Both networks have the same layers, 2-64-64-1-64-64-2. Left, no nonlinearity: the reconstructions lie on a straight line, mean squared error 0.107. Right, tanh: the reconstructions follow the arc, mean squared error 0.002

  • Same layers and units in both; only gg differs. No gg: a flat subspace, however many layers. With gg: a curved manifold

Computed: toy data, 300 points on a noisy half circle

Try it: compress a face

Sparse coding: reconstruct from a few basis functions

The same reconstruction loss, plus L1 regularization on the code, of strength λ\lambda. Unlike L2, L1 sets small values to exactly zero, so no bottleneck is needed, even with more functions than pixels (overcomplete):

ℓ=12 ∥x^−x∥2+λ ∥h∥1\ell = \tfrac12\,\|\hat{\mathbf{x}} - \mathbf{x}\|^2 + \lambda\,\|\mathbf{h}\|_1

One 16 by 16 whitened image patch, shown to be approximately equal to the sum of three learned basis functions, each an oriented stripe pattern multiplied by its coefficient, plus a few more basis functions. To the right, a stem plot of the patch's 256 code coefficients, almost all exactly zero

  • Natural image patches, not faces: a few of 256 learned basis functions per patch. They resemble V1 simple-cell receptive fields

L1 against L2 on the same patch

Two stem plots of the code of one patch, each on its own axis. Left, the L2 code: all 256 coefficients are not zero, the largest about 2. Right, the L1 code, with the same reconstruction quality: only 9 of 256 are not zero, the largest about 8

  • Same patch, same basis functions, same R2R^2 (0.69): L2 uses all 256, L1 uses 9
  • L1 also matters for learning the basis functions: sparseness makes them localized and oriented
  • V1 is overcomplete: about 25 output fibers per LGN input in cat, about 50 in macaque

From reconstructing to generating

An autoencoder reconstructs the image it is given. Can it make a new one?

What if we keep only the decoder?

Two rows of eight square faces from the decoder of a plain autoencoder. Top row, decoded from the codes of real held-out faces: blurred but natural faces. Bottom row, decoded from random codes: faces with severe distortion, grainy and with harsh black and white areas

  • An autoencoder needs an input. Yet you can imagine a face with nothing in front of your eyes
  • So drop the encoder and give the decoder a random code. It gives distorted faces: a random code is unlike the codes the decoder saw in training

Computed: same faces, a larger autoencoder, 1024–512–32–512–1024

Computed on CelebA

Variational autoencoder: sampling new images

Half of a variational autoencoder drawn unit by unit. A held-out face x feeds a column of input units standing for 1024 pixels, then weights to 512 hidden ReLU units, then two columns of 32 circles, the means and the spreads for this face. Those feed a box where a random epsilon makes one draw z, the 32-unit green code. To the right, a small decoder trapezoid and the reconstructed face

  • A variational autoencoder (VAE) adds a loss term keeping codes near a Gaussian: we can sample them
  • Encoder output: 32 means, 32 standard deviations. The code z\mathbf{z} is a random sample
  • Plain backprop is not enough: no error passes back through a random draw. Variational inference gives the loss; rewriting the sample lets backprop train it ("Going further")

What the VAE gives you: new faces

Three rows of square faces. First row, a plain autoencoder decoding random codes: recognizable but distorted, grainy, overly dark and light faces. Second row, a variational autoencoder decoding the same codes: more coherent, blurry faces. Third row, a walk between two held-out faces in the code of the variational autoencoder: eight faces changing gradually

  • Top two rows, the same eight random codes as before. Plain autoencoder: distorted, harsh faces. VAE: more coherent new faces, still blurry and grainy
  • Walk between two codes: take the encoder's mean codes for two faces, za\mathbf{z}_a and zb\mathbf{z}_b, then decode points on the straight line between them, z(t)=(1−t) za+t zb\mathbf{z}(t) = (1-t)\,\mathbf{z}_a + t\,\mathbf{z}_b, with tt from 0 to 1. One face turns gradually into another
Computed on CelebA

Try it: walk the code

Beyond the VAE

  • Extensions of the VAE: many, and you may hear about the β-VAE, which weights the Gaussian term more heavily
  • Alternatives for generation: GANs, diffusion models
  • Image and video generation is an active field. It is less directly relevant to neuroscience, and we skip it today to save time
  • If you want to learn more about generative models, they make a good final project

From reconstruction to prediction

Self-supervised learning: first match views of an image (SimCLR, DINO), then predict what is hidden (MAE). The goal: approach the accuracy of a fully supervised network, without its labels

Equal reconstruction, unequal features

From Balestriero and LeCun: two ResNet-34 autoencoders give near-identical reconstructions of stereos and equal reconstruction loss, yet test accuracy with a linear readout is about 60 against 87 percent
  • Good reconstructions do not imply good features: two autoencoders with the same reconstruction loss score about 60% and 87% on a linear readout (left; the better one also got a supervised signal)
  • So copying the input does not guarantee features useful for other tasks. What else could the target be, with no labels?

Self-supervised learning

  • Self-supervised learning: learning without human labels, for example by matching two views of one image or filling in hidden image regions
  • Pretext task: the training task used to learn the features, such as predicting an image's rotation. Keep the encoder, the network that produces the features; throw away the training head on top of it
  • Pretraining: this initial learning, before adaptation to a particular task. The encoder can then serve new tasks in two ways:
  • Linear readout: freeze the encoder, so that its weights no longer change, and train a linear classifier on its features for the new task
  • Fine-tuning: keep adjusting the pretrained weights for the new task

Early pretext tasks

One cat photograph transformed for four tasks. Rotation: the photo turned 90 degrees, predict the angle. Jigsaw: its nine tiles shuffled, predict the shuffle. Relative position: two neighboring patches, predict where the red one lies relative to the blue. Colorization: the gray-level photo, predict the colors

  • Each is ordinary softmax classification on a label we make from the image: rotation, 4 classes; relative position, 8; jigsaw, which of a chosen set of shuffles. To say which way is up, a network must know something about cats. Useful features, but well below supervised ones

Three classes of methods

Schematic from the Annual Review chapter: two views of a cat image, with representations pulled together and other images pushed apart

Contrastive (SimCLR): match two views, push others apart

Schematic from the Annual Review chapter: one view of a cat through an encoder predicts the output of a running-average copy on another view

Self-distillation (DINO): match a slowly updated copy of itself

Schematic from the Annual Review chapter: a cat image with most patches hidden goes through an encoder and decoder and is compared with the original

Masked prediction (MAE): fill in hidden patches

  • We take them in turn, and for each: what prevents collapse, giving every image the same output, where it excels, where it fails
Serre Annu Rev Vis Sci (in submission), Fig. 2c–e · Chen et al. ICML 2020 · Caron et al. ICCV 2021 · He et al. CVPR 2022

Attraction: two views of one image should agree

One cat photograph and five random views of it: a crop, a crop and flip, a color-jittered version, a gray-level version and a blurred version

  • A view: a copy changed by data augmentation, a stand-in for changes in viewpoint or lighting
  • Over hundreds of epochs, each image gets new random views every time, never one fixed pair
  • What the views change is what the network learns to ignore: position for crops, color for color jitter
  • Attraction alone allows collapse: every image gets the same output, and nothing is learned

Add repulsion: contrastive learning (SimCLR)

One crop of a cat on the left, seven candidates on the right: a koala, a dog, a car, another crop of the same cat outlined in red, bananas, a chair and a teapot. Bars show a schematic probability for each, 0.86 for the other cat crop and at most 0.06 for the others

  • Positives: two random views of one image. Negatives: the other images in the batch, sampled at random. Each view must identify its positive among them: softmax classification
  • The loss, InfoNCE ("Going further"), pulls positives together and pushes negatives apart. Repulsion prevents collapse. SimCLR applies the loss on a small training head after the encoder, thrown away after training

SimCLR: where it excels

Horizontal bar chart of ImageNet accuracy of a linear readout when training uses one kind of augmentation: rotate 2.6 percent, blur 2.6, color 18.8, crop 33.1, and crop plus color together 56.3
  • Linear readout on a frozen ResNet-50: 69% on ImageNet, against 60 to 64% for earlier methods
  • Fine-tuning with few labels: pretrain with no labels, then fine-tune on 1% of the labels. Top-5 accuracy 75%, against 48% for supervised training on that same 1%
  • It depends on the views (figure). No single augmentation is enough. Crop plus color: 56%
Chen et al. ICML 2020 · Fig. 5, replotted

SimCLR: where it fails

Schematic from Chuang et al.: a golden retriever, an augmented view of it as the positive, and negatives: a cat, an airplane and a second golden retriever outlined in red as a false negative
  • False negatives: negatives are sampled at random, with no labels, so two images of one class, such as two dogs, get pushed apart. With 30 negatives on CIFAR-10: about 80% accuracy; 88% when sampled from different classes
  • Batch size: the batch supplies the negatives, and a larger batch gives more of them, including harder ones. Default 4096 images; smaller batches can come close

Second class: self-distillation

DINO: match a slowly updated copy of the network, with no repulsion

DINO: drop repulsion, match a slowly updated copy

Schematic from the Annual Review chapter: one view of a cat goes through the encoder, another view goes through a dashed box labelled encoder, running-average copy. The first output is trained to predict the second, the target, compared in features

  • How DINO avoids collapse with no repulsion: the network learns to match a slowly changing copy of itself that sees another view of the image. The copy's output is adjusted so that it can neither pile onto one value nor flatten out: centering stops one value from dominating, sharpening stops the output from going flat
  • Without repulsion there are no negatives, so no false negatives
Caron et al. ICCV 2021 · Serre Annu Rev Vis Sci (in submission), Fig. 2e · Grill et al. NeurIPS 2020

What a ResNet learns with no labels

Springer spaniel, church, gas pump, tench and parachute photographs, each with first-PC and center-seed cosine maps from a DINO and a supervised ResNet-50. The maps are patchy and do not reliably cover the object
  • The last layer of a DINO ResNet-50 is a 25 × 25 map, one feature vector per location
  • First PC: PCA over one image's locations: its largest contrast in features
  • Cosine to a seed: which locations have feature vectors pointing the same way as at one point
  • With no labels, the maps look much like those of a ResNet-50 trained on 1.3 million labeled images. In both, similar-looking locations group together, without reliably covering the whole object
Computed on five ImageNet photographs · Caron et al. ICCV 2021 · He et al. CVPR 2016 · Geirhos et al. ICLR 2019

The same objective on a transformer

Figure 1 of Caron et al.: photographs with the self-attention map of a DINO vision transformer beneath each, outlining the object with no label, mask or box given

  • The same DINO objective on a vision transformer, an architecture you meet in a later lecture. The map is the last layer's self-attention
  • These maps outline objects more sharply. Not a controlled comparison with the ResNet maps: other photographs, another kind of map

DINOv2: the same parts, matched across images

Figure 1 of Oquab et al.: four columns of photographs with PCA maps of patch features beside them, colored by the first three components. Eagles and airplanes, elephants and a statue, horses and a drawing, vehicles: corresponding parts share colors

  • DINOv2 (Meta): DINO plus masking. Some patches of the network's input are masked, and it must predict what the copy, which sees them, outputs there. 142 million curated images, no labels
  • PCA on the patch features of each column's images, first three components as colors. The same parts get the same colors across poses, styles and even different objects: wings of an eagle and of a plane

DINOv3: sharp features at high resolution

Left, Figure 3 of Simeoni et al.: a market stall of fruit surrounded by cosine similarity maps, each lighting up the patches like one marked patch. Right, Figure 1c and d: PCA maps of a cat and of aerial images, showing fine detail

  • DINOv3 (Meta): 1.7 billion images, 6.7 billion parameters, no labels. One frozen encoder serves many tasks
  • Left: cosine to a seed, as on the ResNet slide, at 4096 pixels: a marked patch finds its own kind of fruit. Right: PCA colors on a cat, and on aerial images from a separate model trained on satellite data
Siméoni et al. arXiv 2025 · Figs. 3, 1c–d

DINO to DINOv3: mostly the same recipe, at scale

DINO (2021) DINOv2 (2023) DINOv3 (2025)
Images 1.3 million 142 million 1.7 billion
Parameters 85 million 1.1 billion 6.7 billion
Added slow copy, centering, sharpening masked patches Gram anchoring
ImageNet readout 80% 87% 88%
  • All with no labels. Mostly more data and larger models, plus fixes that keep training stable at that size. New in the objective: masking in DINOv2, a term for patch similarities in DINOv3
  • Closely related: JEPA (LeCun, also Meta) predicts a slow copy's features for hidden regions

Third class: masked prediction

DINOv2 predicts hidden features; MAE reconstructs hidden pixels. No views to match

Masked autoencoder (MAE): drop the views

Schematic from the Annual Review chapter: a cat image with most patches hidden goes through an encoder and a decoder; the output is compared in pixels with the original

  • MAE (Meta): hide 75% of the patches and reconstruct them, inferring what was not shown
  • A constant output cannot reconstruct different images, so collapse is not the risk. What it adds: no paired views to match
He et al. CVPR 2022 · Serre Annu Rev Vis Sci (in submission), Fig. 2d

MAE: real reconstructions

Bus, vase, zebras, mushrooms and vegetables reconstructed from 75, 85 and 95 percent hidden patches. Reconstructions blur as masking rises, and at 95 percent the green pepper disappears
  • From 25% of the patches, a masked autoencoder (a transformer) reconstructs the bus and the zebras
  • At 95% hidden, it still draws a plausible bus. But the green pepper is gone: filled in from what is typical, not from what was there

SimCLR, DINO, MAE: each removes a dependence

Attraction + repulsion
(SimCLR)
Attraction only
(DINO)
Masked prediction
(MAE)
Removes labels negatives views
Prevents collapse by repulsion slow copy, centering, sharpening targets differ image to image
Excels few labels frozen features: 78% against 68% fine-tuning: 83.6% against 82.8%
Fails false negatives; needs the right views delicate balance frozen readout

On the same transformer (ViT-B/16), with each paper's recipe, DINO gives stronger frozen features; MAE gains more from fine-tuning

One recipe: pretrain once, read out cheaply

Figure 1a of Simeoni et al. 2025: ImageNet accuracy of a linear readout by year, 2012 to 2025. Supervised models rise from about 60 to near 90 percent. Self-supervised models start near 38 percent around 2018 and rise steeply, reaching the same level by 2025
  • Pretrain once with no labels, then fit a linear readout on the frozen encoder
  • Figure: that readout's ImageNet accuracy by year. SL: supervised. WSL: weakly supervised, from hashtags or captions. SSL: self-supervised
  • Self-supervised pretraining now approaches supervised models on ImageNet: labels are needed only for the readout
  • Foundation models work this way: large models pretrained on broad data, typically without labels, then adapted to many tasks

Back to the scaling figure

Scatter plot of ImageNet multilabel accuracy, 60 to 97.5 percent, against training computation in floating-point operations from 10 to the 18 to 10 to the 24, for 40 image encoders released between 2012 and 2025. Accuracy rises from AlexNet near 62 percent to a cluster inside the gray band of trained human annotators, about 92 to 97 percent. Points are colored by objective: supervised black, self-supervised blue, image-text orange, video yellow; circles are CNNs, diamonds transformers, a square the MambaVision hybrid; gray markers are the 2012 to 2018 models. DINOv3, self-supervised, and several image-text transformers sit at the top right, inside the human band. Qwen3-VL is plotted separately at right because its training computation is unpublished.

  • DINOv3 pretrained on 1.7 billion images with no category labels. Others at the top use captions, or labels
  • Few labels are needed afterwards: ten labeled images per ImageNet category, under 1% of the labels, give a linear readout of DINOv2 or DINOv3 about 87% accuracy (any valid label counts). Most of the learning happens in pretraining, without category labels
Serre Annu Rev Vis Sci (in submission) · Siméoni et al. arXiv 2025 · Conwell et al. Nat Commun 2024

Does the brain need labels?

We learned useful features with no labels, from images and video. How do these models compare with biological vision?

Which of these methods is brain-like?

  • Probably none of them. SimCLR, DINO and masked prediction are not proposals for what brains do
  • Why study them, then? They train at scale, and at convergence they give clean, somewhat idealized representations: what a learning principle can produce from images or video
  • It is less about the algorithm than about the principle. Because these algorithms work well, we can vary a principle and test it: the objective, the training data, the constraints on the circuit
  • The test is whether the resulting representations reproduce neural and behavioral data. We are not there yet: there is a long way to go
Serre Annu Rev Vis Sci (in submission)

More accurate is not reliably more brain-like

Five scatter plots in a row of ImageNet multilabel accuracy against alignment with biological vision for 363 classifiers. IT prediction on three datasets: correlation 0.37 for Arcaro, 0.66 for TVSD, minus 0.08 for Triple-N, all well below each measure's ceiling. Human feature use: minus 0.77. Human similarity judgments: minus 0.49. DINOv3 and Qwen3-VL are marked

  • IT prediction: mixed across datasets, well below the ceiling. Human feature use and similarity judgments: alignment falls as accuracy rises
  • A stronger test: do models reproduce how controlled changes in experience alter vision?
Serre Annu Rev Vis Sci (in submission), Fig. 4

From images to video

Toward more ecologically valid data: a continuous stream of experience

From static images to a stream of experience

Five pairs of images of one cat, input on the left and target on the right. Reconstruct the input: the photo and itself. Predict what is hidden: a masked photo and the original. Match two views of one photo: two different crops. These three are bracketed as static images, what most models learn from. Match views close in time and predict the next view: slightly shifted frames, bracketed as a stream, what animals learn from

  • Each step changes what the network must predict: from one image, to two views, to what comes next
  • Most models learn from unrelated images; animals learn from a continuous stream. The main difference is less the amount of experience than its kind
Serre Annu Rev Vis Sci (in submission) · Photographs: THINGSplus

Learning from time: one child's head camera

Head-camera video frames stacked along a time arrow. A group showing a white cat on a green cushion is labelled Class A, and a later group showing a living room with a couch is labelled Class B. The depth of each stack is the segment length
  • The data: about 200 hours of video from a camera on one child's head, between 6 and 32 months. No labels
  • The task: cut the video into segments about 5 minutes long and train a convolutional network (MobileNetV2) to say which segment a frame came from
  • Time alone supplies the training labels. It works on that child's own objects: better than a network trained with ImageNet labels. On new toys, the ImageNet network wins

Object constancy depends on temporal experience

Change how views follow each other in time, and tolerance changes:

  • Monkeys: swapping an object's identity across eye movements reshaped the position tolerance of IT neurons
  • Rats: reared with temporally shuffled movies, they had V1 neurons with less position tolerance, while orientation tuning was largely preserved
  • Newborn chicks: slow rotation of one object supported recognition from new viewpoints; faster rotation favored familiar views

These are among the few neural laws of visual learning established so far. A model of temporal learning must reproduce them

Where matching views fails, and what prediction adds

  • Common fate (parts that move together belong together), a failure: chicks reared with one moving object recognized it on new backgrounds. Self-supervised models trained on views or successive frames, in virtual versions of the same chambers, did not
  • One possible explanation: matching whole views does not by itself require separating an object from its background
  • Prediction, a candidate: a model predicting the next frame segmented objects by motion, and these segments taught a second pathway to see objects from form alone, in static images. Disabling prediction abolished the transfer

Scaling has so far increased the amount of experience models receive, not its kind. A project question

Garimella et al. ICLR 2024 · Ashok et al. arXiv 2026 · Serre Annu Rev Vis Sci (in submission)

Recap

  • Autoencoder: reconstruct the input through a bottleneck. With no nonlinearity, it recovers PCA's subspace. Variational autoencoder: a generative model; sample a code and decode it. Sparse coding: a few active units through L1 regularization
  • Self-supervised learning: derive training targets from the input. Attraction + repulsion (SimCLR), attraction only (DINO), masked prediction (MAE): each removes a dependence. Pretrain once, then fit a linear readout. Video moves this toward more ecologically valid experience
  • From images to video: most models learn from unrelated images; animals learn from a continuous stream. Next lecture: how these models compare with brains
  • Three ways to get a representation close Theme 2: PCA, supervised backprop, and learning without labels

Looking ahead

  • Next lecture, Tuesday: first, how these models compare with brains. Then recurrent networks: Theme 3 opens with networks that keep a state over time, and a network that decides the way a monkey does
  • Mon 11/9: the third assignment is due. Tue 11/10: the fourth is released
  • Thu 11/19: project teams on Canvas. Tue 11/24: project proposal, one page per team. Whether a network trained without labels predicts neural responses in macaque visual cortex (TVSD, Triple-N) as well as a supervised one is a project
  • Tue 12/1: Exam 2, on the second half of the course, including today

Further reading for the research trail

Optional. Any paper from any lecture qualifies; these are preferred.

Minute paper

Submit on Canvas → Minute papers → Minute Paper 15 (access code read out in class).

Write three brief points in your own words:

  1. Pick one method from today (autoencoder, SimCLR, DINO or masked prediction). In your own words: what is the network trained to predict, and where does that target come from, if not from a human label?
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Going further

The VAE loss and the contrastive loss in equations and code. More on visual experience and video

The VAE loss

ℓ=12σx2 ∥x^−x∥2⏟reconstruction  +  β  12∑j=1k(μj2+σj2−log⁡σj2−1)⏟KL divergence to a standard Gaussian\ell = \underbrace{\tfrac{1}{2\sigma_x^2}\,\|\hat{\mathbf{x}} - \mathbf{x}\|^2}_{\text{reconstruction}} \;+\; \beta\;\underbrace{\tfrac12\sum_{j=1}^{k}\big(\mu_j^2 + \sigma_j^2 - \log\sigma_j^2 - 1\big)}_{\text{KL divergence to a standard Gaussian}}

  • The reconstruction term is the loss of the plain autoencoder. The KL term is the Gaussian constraint: it is zero when every μj=0\mu_j = 0 and σj=1\sigma_j = 1. A plain VAE has β=1\beta = 1; a β-VAE has β>1\beta > 1
  • σx2\sigma_x^2: the fixed pixel-noise variance. ⊙\odot: elementwise product. Reparameterization: write the sample as z=μ+σ⊙ϵ\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon}, with ϵ∼N(0,I)\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0},\mathbf{I}). All the randomness sits in ϵ\boldsymbol{\epsilon}, so the error passes back through μ\boldsymbol{\mu} and σ\boldsymbol{\sigma} by ordinary backprop

The InfoNCE loss

Here z(i)\mathbf{z}^{(i)} is the head's output for view ii, not a VAE sample. Cosine similarity s(u,v)=u⊤v∥u∥ ∥v∥s(\mathbf{u},\mathbf{v}) = \dfrac{\mathbf{u}^\top\mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|} and temperature τ\tau. For a positive pair (i,j)(i,j):

ℓ(i,j)=−log⁡exp⁡ ⁣(s(z(i),z(j))/τ)∑r≠iexp⁡ ⁣(s(z(i),z(r))/τ)\ell^{(i,j)} = -\log\frac{\exp\!\big(s(\mathbf{z}^{(i)},\mathbf{z}^{(j)})/\tau\big)}{\sum_{r\neq i}\exp\!\big(s(\mathbf{z}^{(i)},\mathbf{z}^{(r)})/\tau\big)}

  • Softmax cross-entropy where the target class is the matching view: a batch of BB images gives 2B2B views, and each view does a (2B−1)(2B{-}1)-way classification
  • Familiar machinery, new label source. The temperature τ\tau sets how hard the closest negatives push

PyTorch: the InfoNCE loss

import torch
import torch.nn.functional as F

def info_nce_loss(z, tau=0.5):
    # z: (2B, d), two augmented views stacked; rows 0..B-1 pair with B..2B-1
    z = F.normalize(z, dim=1)                 # cosine similarity = dot of unit vectors
    sim = z @ z.T / tau                       # (2B, 2B) similarity matrix
    n = z.shape[0]
    sim.fill_diagonal_(float("-inf"))         # a view is never its own positive
    targets = (torch.arange(n) + n // 2) % n  # the positive of row i is its other view
    return F.cross_entropy(sim, targets)      # softmax cross-entropy
  • Cross-entropy over the batch: the positive view is the target class
  • The supervised toolkit, cross-entropy and backprop, unchanged; only the label source changed

How much visual experience?

  • Models: on the order of 10⁹ image presentations in pretraining. Children: 10⁸ to 10⁹ visual samples by age six, up to about 4 × 10⁹ by early adulthood
  • Children are still more efficient. Networks needed more examples than estimated childhood exposure to match three-year-olds' recognition under degraded viewing
  • Biological vision does not start from scratch. Distinct visual pathways are detectable in newborns, spontaneous retinal activity before eye opening has structure related to motion, and inexperienced chicks already prefer biological motion
  • Pretraining may stand in for some of what evolution and prenatal development provide

Three learning signals in a stream

Signal What it is What it can teach
Temporal continuity an object persists across successive views object constancy: recognizing an object despite changes in its retinal image
Common fate parts that move together belong together which regions form one object
Predictability one view regularly follows another what causes change: moving objects and their interactions
  • Each signal is available in the stream, with no labels. Invariance alone is not enough: a representation that treated every image alike would be invariant too, so it must stay selective for object identity
Serre Annu Rev Vis Sci (in submission) · Sevilla-Lara et al. WACV 2021

Models that learn from video and generate it

  • Learning from video: V-JEPA (Meta) predicts the features of hidden parts of a clip, the masked idea extended to time
  • A stream is not enough: in action-recognition benchmarks, many categories can be recognized from a single frame, so a model need not use time at all
  • What is predicted may matter: with the same input, predicting subsequent views brought feature use closer to people's more consistently than filling in between views
  • Generating video: models such as Veo (Google) synthesize video from text. Exciting, and still a good final project. These data move models toward more ecologically valid experience
Bardes et al. TMLR 2024 · Sevilla-Lara et al. WACV 2021 · Zhou et al. 2026 (our lab) · Serre Annu Rev Vis Sci (in submission)

- Last lecture of Theme 2. Next time: recurrent networks - Today: reconstruct the input (autoencoder, sparse coding), generate new samples (variational autoencoder), match views and predict what is hidden (self-supervised learning), then video and experience

- Figure 1 of my Annual Review chapter (in submission). Linear readout on frozen features, ImageNet multilabel accuracy: a prediction counts if it matches any valid label (Shankar et al. 2020) - Vertical scale is logarithmic in error, so it expands differences near 100% - DINOv3-7B: about 1.4 × 10²³ operations. AlexNet: about 4.6 × 10¹⁷. Ratio about 300,000; the plot spans about six orders of magnitude, a factor of a million - AlexNet trained 5 to 6 days on two GPUs, hence about 4,500 years. Compares workloads at a fixed rate; DINOv3 could not really be trained on 2012 hardware - Gray band: trained annotators with a guide, about 26 s per image. Not a ceiling on human recognition - Architecture, data and objective vary together: a trend, not a scaling law

- Same figure, read by color (objective) and symbol (architecture). Black supervised, blue self-supervised, orange image–text, yellow video; circles CNNs, diamonds transformers - Captions are a weaker, far more plentiful supervision than category labels - Readouts used to evaluate these models still use labels - Transformers in a later lecture

- Supervised: photographs labeled "cat" or "dog"; weights change to shrink the error on the labels - Unsupervised: e.g. group photographs that look alike, never told the categories - Most of what a visual system sees has no category label. That does not mean no teaching signal at all

- PCA: the top k directions give the best k-dimensional reconstruction - Autoencoder: also reconstruction, written as a loss, with weights found by gradient descent

- Drawn unit by unit: every line is a weight, and the network is an ordinary multilayer perceptron trained by backprop. Only 12 of the 1024 input and output units are drawn - The network is trained to copy its input, which sounds useless. What makes it useful is the narrow middle: 1024 pixels must pass through 32 numbers, so the code has to capture what faces have in common - Real computation: a face the network never saw, its 32 code values (darker green = more active; a ReLU code can set units to zero, though on this face all 32 are active) and the trained network's output. On 2,000 held-out faces: median R² 0.81. PCA with 32 components: 0.82 on the same faces. This one-layer network does not beat PCA - Encoder also names the part of any trained network before its readout, as in the explainability lecture and the assignment, where you froze a ResNet-50's encoder

- ℓ: loss for one input. ℒ: average over N training inputs - The ½ only cleans up the gradient: the gradient of ½‖e‖² with respect to x̂ is e, where e = x̂ − x - PCA, autoencoders and sparse coding: all unsupervised

- x is one point in a 1024-dimensional space, one axis per pixel. h is where that point lands along 32 directions - The dot product of a face with a row is large when the face looks like the row - Training picks the rows; nobody chose them - In the PCA lecture: projected variance and squared reconstruction error add to a fixed total, so maximizing one minimizes the other

- PCA finds the best k-dimensional reconstruction in closed form. At its global minimum, a linear autoencoder spans the same subspace - V_k has orthonormal columns; W1 and W2 need not. The result holds at the global minimum, for centered data, when the k-th and (k+1)-th eigenvalues differ - Which A you get depends on the initialization: compare seeds in the demo

- Figure: 60,000 training and 2,000 held-out faces. Linear network fitted by alternating least squares to full convergence. Median held-out R² 0.815 for both; largest angle between subspaces under 0.001° - With backprop (Adam) from three random seeds: same R² to three decimals, median angle under 1°, largest angle 6° to 7° after the training budget, and different rows. The slowest directions converge slowly. See the demo

- A curve in 2-D needs one number: position along it. Intrinsic dimension, from the t-SNE and UMAP lecture - Both networks: 2–64–64–1–64–64–2, one code unit, Adam, 6,000 steps, same seed. Right: tanh after each hidden layer, linear code and output - Left: reconstructions on one line, PC1 to within 0.01°. Error falls from 0.107 to 0.002 when only g is added - Without g, a stack of layers multiplies out to one matrix, so extra layers and units do not help

- Reducing k compresses the face. The linear autoencoder's reconstructions match PCA's at every k - Drag k from 128 down to 1. The face loses identity, then expression, and ends near the mean face - Change the seed: other rows of W1, same reconstruction

- Hubel and Wiesel measured V1's oriented templates, not where they come from - Figure, computed: one 16 × 16 patch of a whitened natural photograph, its three largest basis functions, its full code. 9 of 256 coefficients are non-zero, R² = 0.69. A negative coefficient flips light and dark. Unlike the encoder before, the code is inferred for each patch by a solver; the basis functions are shared across patches. About 7 per patch on average - ‖h‖₁ is the sum of |hⱼ| - L1 has a kink at zero, so codes come from solvers built for it: ISTA, a gradient step and then a shrink of each value (Beck and Teboulle 2009), or least-angle regression, used here - 256 functions for 256 pixels is a complete code; Olshausen and Field (1997) went overcomplete - Against 93 macaque simple cells (Ringach 2002): about as narrow, about twice as long. A limited match. 30,000 whitened patches of 500 photographs

- L2 pulls in proportion to a value. L1 pulls equally at any size, so small values reach exactly zero - The two plots have different vertical scales: L2's largest coefficient is about 2, L1's about 8 - Our computation: basis functions fixed, already learned with L1. Only the code changes, with the L2 strength chosen so that this patch's R² matches the L1 code's - Learning is a separate point: Olshausen and Field (1996) obtained localized, oriented functions by asking for a sparse code. PCA on natural images, with no sparseness, gives global patterns - The 25:1 count: axons from layers 2/3 to higher areas against LGN inputs (Olshausen and Field 2004). Macaque, about 50:1 (Olshausen and Field 2006). A trade-off: more neurons cost more, and an extreme sparse code becomes one neuron per event - Our dictionary is complete, 256 functions for 256 pixels

- So far: compress and reconstruct. Next: generate new samples that follow the same distribution as the training data

- Same faces and split as the earlier slides; plain autoencoder 1024–512–32–512–1024, the same layers as the VAE on the next slides. Random codes from a standard Gaussian, first eight draws, no selection. Nothing in the plain autoencoder's loss makes its training codes follow that distribution

- Autoencoder: an image goes in, the same image should come out. Generative model: a random code goes in, a new image comes out - The added term is a KL divergence between each image's code distribution and a standard Gaussian. It makes sampling from a standard Gaussian more reliable; it does not guarantee a good image everywhere, because it competes with reconstruction - We minimize the negative of the evidence lower bound (ELBO): reconstruction cost plus that KL term - Details in "Going further" - For the exam: what the VAE adds to the autoencoder and why; not the math

- Same faces and split as the autoencoder and PCA slides. Both networks 1024–512–32–512–1024. First eight random codes, no selection - Over 512 random codes, distance to the nearest training face (pixels, L2): 20.8 plain, 5.4 VAE. Held-out median R²: 0.82 plain, 0.84 VAE - Blur: the decoder must be right on average over nearby codes, and an average of slightly different faces is a blurred face

- The same Gaussian code gives different faces from the plain autoencoder and the VAE - Press Random code, then switch between the two models: the same draw goes through each decoder - Choose a face and move one unit at a time. Does one unit change one property? Units need not line up with named properties - Walk: a slider moves the code in a straight line between two faces

- One link to neuroscience, for the research trail: single β-VAE units match single face-patch neurons (Higgins et al. 2021) - GAN: a generator trained against a critic that tells real from generated. Diffusion: a network that removes a little noise, run many times

- So far the target was the input itself. Now the target comes from the data but is withheld from the network

- Left: plain autoencoders on Imagenette, not masked ones. Reconstruction favors high-variance, coarse content; low-variance detail supports recognition (Balestriero and LeCun 2024). TinyImageNet: top 90% of the variance 45%, bottom 20% 55% - Next word in a text is the same move for language: later lecture

- Readout: the linear readout from the geometry lectures; papers say linear probe - Encoder: the part before the readout, as when you froze a ResNet-50 in the assignment

- Doersch et al. (2015) relative position: which of 8 neighbors is this patch? Noroozi and Favaro (2016) jigsaw: 9 tiles shuffled by one of a chosen set of permutations, such as 100; which one? Gidaris et al. (2018) rotation: which of 4 angles? Zhang et al. (2016) colorization: per pixel, which of the quantized color bins? - All are trained like any classifier: softmax, cross-entropy, backprop. Only the label source differs - Shortcuts exist: a horizon gives away rotation with no knowledge of objects

- These replaced the earlier pretext tasks because their features are far better

- Two views of one photograph: a positive pair - The only "label" is which photograph a view came from - Each epoch redraws the augmentations for every image, so one image is matched across hundreds of different pairs of views - Crops and color changes stand in for what an animal gets by moving through the world - Every method below has to deal with collapse

- Negatives: all other images in the batch. B images give 2B views, each a (2B − 1)-way classification - The head is thrown away because the features before it transfer better. The head discards what differs between views: on ImageNet, rotation is decoded at 68% from the encoder, 26% from the head, chance 25% (Chen et al. 2020). Probabilities in the figure are illustrative - Equation and PyTorch at the end of the deck

- ImageNet top-1, except the 1% result, which is top-5: correct class among the five most likely - Figure: five cells of Chen et al.'s Fig. 5, replotted. Only one of the two views is transformed, not the full recipe, so numbers sit below 69% - Why crop plus color: two crops share a color histogram, so the network could match on color alone

- Figure: the second retriever is a false negative. x⁺ the positive, xᵢ⁻ the negatives - False negatives are rarer with more classes; this measurement is on CIFAR-10, not full ImageNet - Batch 256 against 8192 at 100 epochs: 57.5% against 64.8%; 62.8% against 64.8% with a better learning rate (Chen et al., Table B.1) - Not specific to SimCLR: invariance to an augmentation can hurt a task that depends on it, such as telling red from yellow cars (Xiao et al. 2021)

- DINO: match a slowly updated copy of the network, with no repulsion

- Both outputs are probability distributions over learned dimensions, not categories - The copy's weights are a running average of the network's own, and no gradient flows through it - Centering subtracts a running mean from the copy's scores. Sharpening uses a low temperature, so its probabilities are more peaked - You used DINO's ResNet-50 in the assignment. Slowly averaged targets came from BYOL (Grill et al. 2020)

- Not an abstract code for object location. Position comes from the layout of the map; the vectors carry appearance. CNNs trained on ImageNet lean on texture (Geirhos et al. 2019); we did not test texture against shape here - Fine edges in the overlay come from the photo underneath - Our computation: layer4 at 800 px, the same two ResNet-50s as the assignment. Five ImageNet photographs, the first of each class in sorted order; seed at the image center; PC sign and color scale set for display. Five photos: description, not evidence

- Caron et al. (2021), Fig. 1: self-attention, last layer of a ViT-S/8 - Attention: later lecture - DINOv2 and v3 add masked prediction

- A vision transformer, so one feature vector per image patch. Background removed by thresholding the first component - Another figure in the paper draws matches between patches of a bus and a toy jeep, or an elephant and a statue

- Same PCA and cosine maps as you saw on the ResNet, now on a very large transformer - Linear readouts on these frozen features also do depth and segmentation (paper's Table 3)

- DINOv3's authors train "following the DINOv2 algorithm at scale", and report that "scale leads to a degradation of dense features". Masked patches: the network predicts the copy's output at patches hidden from it. Gram anchoring fixes the degradation: patch-to-patch similarities are held close to those of an earlier version of the network - DINOv2 also has a regularizer that spreads features and a short high-resolution phase at the end. DINOv3 also has new position embeddings and distills the large model into smaller ones - Readouts: DINO ViT-B/8, from its paper. DINOv2 ViT-g/14 and DINOv3 ViT-7B/16, from the DINOv3 paper's comparison. Evaluation details differ, and the DINOv2 paper itself reports 86.5%

- MAE: no views to match. Hide most of an image and fill it in

- Few patches hidden: fill in from neighbors. Most hidden: must use what it knows about objects and layout - MAE's encoder sees only the visible 25%, so it is cheap to train - With no augmentation at all: fine-tuned 84.0, linear 65.7, against 84.9 and 73.5 with crops (ViT-L) - Language hides words; next-word prediction is a blank at the end: later lecture

- He et al. (2022), Fig. 4, a transformer. Convolutional version: SparK (Tian et al. 2023), ConvNeXt at 60%. - The paper calls these plausible; the pepper is our reading

- How each builds its target, and how each avoids constant outputs - Transformer numbers in the Excels row. Protocols differ across papers - DINOv2 and v3 combine self-distillation with masked prediction

- ImageNet linear readout on frozen features, each point the best method of its year as reported in the paper, largest model - On the same ResNet-50: DINO and SwAV 75%, SimCLR 69%, supervised 79% (Caron et al. 2021, Table 2)

- Foundation models: pretrained without labels, adapted with a small readout - The 87% is not plotted: separate evaluation on the same models, from the chapter - Conwell et al. (2024): training images mattered more than architecture or objective - Project seed: fix the architecture, vary the objective (labels, DINO, image–text). Which predicts TVSD or Triple-N better, at which block? The three ResNet-50s from the assignment are a start

- These models learn without labels. How do they compare with biological vision?

- These are training procedures, not mechanisms: nobody proposes that cortex masks 75% of an image or keeps a slow copy of itself - The next slides compare representations, not algorithms

- Correlational by construction: models differ at once in architecture, objective, data and scale - TVSD is the exception, with continued improvement; the reason for the difference between datasets is open - More accurate models may use different visual strategies: local texture where people use parts or 3-D structure

- Toward more ecologically valid data: a continuous stream of experience

- Left to right: each step changes what the network must predict. Stream frames are a sliding crop, not video

- A bit of a hack: nobody cares which five minutes a frame came from. It works because objects and scenes change slowly - MobileNetV2 from scratch, SAYCam (Sullivan et al. 2021), one model per child. Reported model also used color augmentation - This classifies segments; it is not contrastive. Their contrastive version, neighbors as positives, did clearly worse - 288 s segments beat 18 s - Readout on that child's 26 categories: about 60% against 54%. New toy exemplars: about 45% against 60%. Models trained on the other two children, tested on this child's labeled frames, do not beat ImageNet. Plots in the paper; it is on the research trail

- Neural law: a relation between a controlled change in experience and a change in neural responses or perception - Position tolerance: a neuron keeps its preference across positions - Li and DiCarlo's monkeys were adults: plasticity, not development; saccade signals not ruled out - Networks on chicks' views reproduce the viewpoint pattern (Pandey et al. 2024); not yet tested on altered experience

- Garimella et al. (2024): virtual rearing chambers matching the chick experiments (Wood and Wood 2021). Models pretrained at scale can learn to separate objects, so this is about what the stream asks for - Ashok et al. (2026), from our lab: a dorsal-like and a ventral-like pathway. Infants also see objects first through common motion (Kellman and Spelke 1983)

- The minute paper still counts. Further reading and "Going further" are optional - Definitions to know for the exam: autoencoder, encoder, decoder, bottleneck, reconstruction loss, sparse code and L1 regularization, generative model, pretext task, linear readout and fine-tuning, positive pair and negatives, collapse, masked prediction, and what the VAE adds to the autoencoder

- The exam covers the main slides of this deck; the further-reading papers are optional

- Before posting, skim the existing entries on your paper; your post must add something not already said

- Submitted on Canvas (Minute papers → Minute Paper 15); credit for a thoughtful attempt

- Optional: the VAE loss; the InfoNCE loss, in an equation and in PyTorch; how much visual experience models and children get; three learning signals in a stream; models that learn from video

- With β = 1, the expected loss over sampled codes is the negative of a lower bound on log p(x), the ELBO, up to a constant. That bound is what variational inference optimizes - σₓ²: the assumed pixel noise variance, 0.01 for our face network. ⊙: elementwise multiplication - The KL line is the closed form for a Gaussian code with diagonal covariance against a standard Gaussian

- z⁽ⁱ⁾: view i's vector from the projection head. This z differs from the VAE's sampled code. The code on the next slide averages this loss over all views. The denominator holds the other view and all negatives - Cosine similarity: dot product over lengths, so only the angle counts - τ scales similarities before the softmax. Small τ: the most similar wrong view dominates - van den Oord et al. (2018); SimCLR's form is NT-Xent

- The same cross-entropy you wrote for the classifier in the assignment, applied to a similarity matrix instead of class scores - With a real encoder in front of it, this loss and a few hundred lines of data augmentation are SimCLR

- Counts depend on what a visual sample is: waking time at 1 to 10 samples per second. Successive views are correlated - Rough numbers, for scale only

- Crops stand in for the views an animal gets by moving - Getting a stream does not mean using these signals: many action-recognition categories can be told from a single frame (Sevilla-Lara et al. 2021)

- Predicting subsequent views against interpolation: Zhou et al. (2026), from our lab. Whether generative video models understand physics is an open question

trail:start