import torch
import torch.nn.functional as F
definfo_nce_loss(z, tau=0.5):
# z: (2B, d), two augmented views stacked; rows 0..B-1 pair with B..2B-1
z = F.normalize(z, dim=1) # cosine similarity = dot of unit vectors
sim = z @ z.T / tau # (2B, 2B) similarity matrix
n = z.shape[0]
sim.fill_diagonal_(float("-inf")) # a view is never its own positive
targets = (torch.arange(n) + n // 2) % n # the positive of row i is its other viewreturn F.cross_entropy(sim, targets) # softmax cross-entropy
Cross-entropy over the batch: the positive view is the target class
The supervised toolkit, cross-entropy and backprop, unchanged; only the label source changed
Models: on the order of 10⁹ image presentations in pretraining. Children: 10⁸ to 10⁹ visual samples by age six, up to about 4 × 10⁹ by early adulthood
Children are still more efficient. Networks needed more examples than estimated childhood exposure to match three-year-olds' recognition under degraded viewing
Biological vision does not start from scratch. Distinct visual pathways are detectable in newborns, spontaneous retinal activity before eye opening has structure related to motion, and inexperienced chicks already prefer biological motion
Pretraining may stand in for some of what evolution and prenatal development provide
object constancy: recognizing an object despite changes in its retinal image
Common fate
parts that move together belong together
which regions form one object
Predictability
one view regularly follows another
what causes change: moving objects and their interactions
Each signal is available in the stream, with no labels. Invariance alone is not enough: a representation that treated every image alike would be invariant too, so it must stay selective for object identity
Learning from video: V-JEPA (Meta) predicts the features of hidden parts of a clip, the masked idea extended to time
A stream is not enough: in action-recognition benchmarks, many categories can be recognized from a single frame, so a model need not use time at all
What is predicted may matter: with the same input, predicting subsequent views brought feature use closer to people's more consistently than filling in between views
Generating video: models such as Veo (Google) synthesize video from text. Exciting, and still a good final project. These data move models toward more ecologically valid experience
- Last lecture of Theme 2. Next time: recurrent networks
- Today: reconstruct the input (autoencoder, sparse coding), generate new samples (variational autoencoder), match views and predict what is hidden (self-supervised learning), then video and experience
- Figure 1 of my Annual Review chapter (in submission). Linear readout on frozen features, ImageNet multilabel accuracy: a prediction counts if it matches any valid label (Shankar et al. 2020)
- Vertical scale is logarithmic in error, so it expands differences near 100%
- DINOv3-7B: about 1.4 × 10²³ operations. AlexNet: about 4.6 × 10¹⁷. Ratio about 300,000; the plot spans about six orders of magnitude, a factor of a million
- AlexNet trained 5 to 6 days on two GPUs, hence about 4,500 years. Compares workloads at a fixed rate; DINOv3 could not really be trained on 2012 hardware
- Gray band: trained annotators with a guide, about 26 s per image. Not a ceiling on human recognition
- Architecture, data and objective vary together: a trend, not a scaling law
- Same figure, read by color (objective) and symbol (architecture). Black supervised, blue self-supervised, orange image–text, yellow video; circles CNNs, diamonds transformers
- Captions are a weaker, far more plentiful supervision than category labels
- Readouts used to evaluate these models still use labels
- Transformers in a later lecture
- Supervised: photographs labeled "cat" or "dog"; weights change to shrink the error on the labels
- Unsupervised: e.g. group photographs that look alike, never told the categories
- Most of what a visual system sees has no category label. That does not mean no teaching signal at all
- PCA: the top k directions give the best k-dimensional reconstruction
- Autoencoder: also reconstruction, written as a loss, with weights found by gradient descent
- Drawn unit by unit: every line is a weight, and the network is an ordinary multilayer perceptron trained by backprop. Only 12 of the 1024 input and output units are drawn
- The network is trained to copy its input, which sounds useless. What makes it useful is the narrow middle: 1024 pixels must pass through 32 numbers, so the code has to capture what faces have in common
- Real computation: a face the network never saw, its 32 code values (darker green = more active; a ReLU code can set units to zero, though on this face all 32 are active) and the trained network's output. On 2,000 held-out faces: median R² 0.81. PCA with 32 components: 0.82 on the same faces. This one-layer network does not beat PCA
- Encoder also names the part of any trained network before its readout, as in the explainability lecture and the assignment, where you froze a ResNet-50's encoder
- ℓ: loss for one input. ℒ: average over N training inputs
- The ½ only cleans up the gradient: the gradient of ½‖e‖² with respect to x̂ is e, where e = x̂ − x
- PCA, autoencoders and sparse coding: all unsupervised
- x is one point in a 1024-dimensional space, one axis per pixel. h is where that point lands along 32 directions
- The dot product of a face with a row is large when the face looks like the row
- Training picks the rows; nobody chose them
- In the PCA lecture: projected variance and squared reconstruction error add to a fixed total, so maximizing one minimizes the other
- PCA finds the best k-dimensional reconstruction in closed form. At its global minimum, a linear autoencoder spans the same subspace
- V_k has orthonormal columns; W1 and W2 need not. The result holds at the global minimum, for centered data, when the k-th and (k+1)-th eigenvalues differ
- Which A you get depends on the initialization: compare seeds in the demo
- Figure: 60,000 training and 2,000 held-out faces. Linear network fitted by alternating least squares to full convergence. Median held-out R² 0.815 for both; largest angle between subspaces under 0.001°
- With backprop (Adam) from three random seeds: same R² to three decimals, median angle under 1°, largest angle 6° to 7° after the training budget, and different rows. The slowest directions converge slowly. See the demo
- A curve in 2-D needs one number: position along it. Intrinsic dimension, from the t-SNE and UMAP lecture
- Both networks: 2–64–64–1–64–64–2, one code unit, Adam, 6,000 steps, same seed. Right: tanh after each hidden layer, linear code and output
- Left: reconstructions on one line, PC1 to within 0.01°. Error falls from 0.107 to 0.002 when only g is added
- Without g, a stack of layers multiplies out to one matrix, so extra layers and units do not help
- Reducing k compresses the face. The linear autoencoder's reconstructions match PCA's at every k
- Drag k from 128 down to 1. The face loses identity, then expression, and ends near the mean face
- Change the seed: other rows of W1, same reconstruction
- Hubel and Wiesel measured V1's oriented templates, not where they come from
- Figure, computed: one 16 × 16 patch of a whitened natural photograph, its three largest basis functions, its full code. 9 of 256 coefficients are non-zero, R² = 0.69. A negative coefficient flips light and dark. Unlike the encoder before, the code is inferred for each patch by a solver; the basis functions are shared across patches. About 7 per patch on average
- ‖h‖₁ is the sum of |hⱼ|
- L1 has a kink at zero, so codes come from solvers built for it: ISTA, a gradient step and then a shrink of each value (Beck and Teboulle 2009), or least-angle regression, used here
- 256 functions for 256 pixels is a complete code; Olshausen and Field (1997) went overcomplete
- Against 93 macaque simple cells (Ringach 2002): about as narrow, about twice as long. A limited match. 30,000 whitened patches of 500 photographs
- L2 pulls in proportion to a value. L1 pulls equally at any size, so small values reach exactly zero
- The two plots have different vertical scales: L2's largest coefficient is about 2, L1's about 8
- Our computation: basis functions fixed, already learned with L1. Only the code changes, with the L2 strength chosen so that this patch's R² matches the L1 code's
- Learning is a separate point: Olshausen and Field (1996) obtained localized, oriented functions by asking for a sparse code. PCA on natural images, with no sparseness, gives global patterns
- The 25:1 count: axons from layers 2/3 to higher areas against LGN inputs (Olshausen and Field 2004). Macaque, about 50:1 (Olshausen and Field 2006). A trade-off: more neurons cost more, and an extreme sparse code becomes one neuron per event
- Our dictionary is complete, 256 functions for 256 pixels
- So far: compress and reconstruct. Next: generate new samples that follow the same distribution as the training data
- Same faces and split as the earlier slides; plain autoencoder 1024–512–32–512–1024, the same layers as the VAE on the next slides. Random codes from a standard Gaussian, first eight draws, no selection. Nothing in the plain autoencoder's loss makes its training codes follow that distribution
- Autoencoder: an image goes in, the same image should come out. Generative model: a random code goes in, a new image comes out
- The added term is a KL divergence between each image's code distribution and a standard Gaussian. It makes sampling from a standard Gaussian more reliable; it does not guarantee a good image everywhere, because it competes with reconstruction
- We minimize the negative of the evidence lower bound (ELBO): reconstruction cost plus that KL term
- Details in "Going further"
- For the exam: what the VAE adds to the autoencoder and why; not the math
- Same faces and split as the autoencoder and PCA slides. Both networks 1024–512–32–512–1024. First eight random codes, no selection
- Over 512 random codes, distance to the nearest training face (pixels, L2): 20.8 plain, 5.4 VAE. Held-out median R²: 0.82 plain, 0.84 VAE
- Blur: the decoder must be right on average over nearby codes, and an average of slightly different faces is a blurred face
- The same Gaussian code gives different faces from the plain autoencoder and the VAE
- Press Random code, then switch between the two models: the same draw goes through each decoder
- Choose a face and move one unit at a time. Does one unit change one property? Units need not line up with named properties
- Walk: a slider moves the code in a straight line between two faces
- One link to neuroscience, for the research trail: single β-VAE units match single face-patch neurons (Higgins et al. 2021)
- GAN: a generator trained against a critic that tells real from generated. Diffusion: a network that removes a little noise, run many times
- So far the target was the input itself. Now the target comes from the data but is withheld from the network
- Left: plain autoencoders on Imagenette, not masked ones. Reconstruction favors high-variance, coarse content; low-variance detail supports recognition (Balestriero and LeCun 2024). TinyImageNet: top 90% of the variance 45%, bottom 20% 55%
- Next word in a text is the same move for language: later lecture
- Readout: the linear readout from the geometry lectures; papers say linear probe
- Encoder: the part before the readout, as when you froze a ResNet-50 in the assignment
- Doersch et al. (2015) relative position: which of 8 neighbors is this patch? Noroozi and Favaro (2016) jigsaw: 9 tiles shuffled by one of a chosen set of permutations, such as 100; which one? Gidaris et al. (2018) rotation: which of 4 angles? Zhang et al. (2016) colorization: per pixel, which of the quantized color bins?
- All are trained like any classifier: softmax, cross-entropy, backprop. Only the label source differs
- Shortcuts exist: a horizon gives away rotation with no knowledge of objects
- These replaced the earlier pretext tasks because their features are far better
- Two views of one photograph: a positive pair
- The only "label" is which photograph a view came from
- Each epoch redraws the augmentations for every image, so one image is matched across hundreds of different pairs of views
- Crops and color changes stand in for what an animal gets by moving through the world
- Every method below has to deal with collapse
- Negatives: all other images in the batch. B images give 2B views, each a (2B − 1)-way classification
- The head is thrown away because the features before it transfer better. The head discards what differs between views: on ImageNet, rotation is decoded at 68% from the encoder, 26% from the head, chance 25% (Chen et al. 2020). Probabilities in the figure are illustrative
- Equation and PyTorch at the end of the deck
- ImageNet top-1, except the 1% result, which is top-5: correct class among the five most likely
- Figure: five cells of Chen et al.'s Fig. 5, replotted. Only one of the two views is transformed, not the full recipe, so numbers sit below 69%
- Why crop plus color: two crops share a color histogram, so the network could match on color alone
- Figure: the second retriever is a false negative. x⁺ the positive, xᵢ⁻ the negatives
- False negatives are rarer with more classes; this measurement is on CIFAR-10, not full ImageNet
- Batch 256 against 8192 at 100 epochs: 57.5% against 64.8%; 62.8% against 64.8% with a better learning rate (Chen et al., Table B.1)
- Not specific to SimCLR: invariance to an augmentation can hurt a task that depends on it, such as telling red from yellow cars (Xiao et al. 2021)
- DINO: match a slowly updated copy of the network, with no repulsion
- Both outputs are probability distributions over learned dimensions, not categories
- The copy's weights are a running average of the network's own, and no gradient flows through it
- Centering subtracts a running mean from the copy's scores. Sharpening uses a low temperature, so its probabilities are more peaked
- You used DINO's ResNet-50 in the assignment. Slowly averaged targets came from BYOL (Grill et al. 2020)
- Not an abstract code for object location. Position comes from the layout of the map; the vectors carry appearance. CNNs trained on ImageNet lean on texture (Geirhos et al. 2019); we did not test texture against shape here
- Fine edges in the overlay come from the photo underneath
- Our computation: layer4 at 800 px, the same two ResNet-50s as the assignment. Five ImageNet photographs, the first of each class in sorted order; seed at the image center; PC sign and color scale set for display. Five photos: description, not evidence
- Caron et al. (2021), Fig. 1: self-attention, last layer of a ViT-S/8
- Attention: later lecture
- DINOv2 and v3 add masked prediction
- A vision transformer, so one feature vector per image patch. Background removed by thresholding the first component
- Another figure in the paper draws matches between patches of a bus and a toy jeep, or an elephant and a statue
- Same PCA and cosine maps as you saw on the ResNet, now on a very large transformer
- Linear readouts on these frozen features also do depth and segmentation (paper's Table 3)
- DINOv3's authors train "following the DINOv2 algorithm at scale", and report that "scale leads to a degradation of dense features". Masked patches: the network predicts the copy's output at patches hidden from it. Gram anchoring fixes the degradation: patch-to-patch similarities are held close to those of an earlier version of the network
- DINOv2 also has a regularizer that spreads features and a short high-resolution phase at the end. DINOv3 also has new position embeddings and distills the large model into smaller ones
- Readouts: DINO ViT-B/8, from its paper. DINOv2 ViT-g/14 and DINOv3 ViT-7B/16, from the DINOv3 paper's comparison. Evaluation details differ, and the DINOv2 paper itself reports 86.5%
- MAE: no views to match. Hide most of an image and fill it in
- Few patches hidden: fill in from neighbors. Most hidden: must use what it knows about objects and layout
- MAE's encoder sees only the visible 25%, so it is cheap to train
- With no augmentation at all: fine-tuned 84.0, linear 65.7, against 84.9 and 73.5 with crops (ViT-L)
- Language hides words; next-word prediction is a blank at the end: later lecture
- He et al. (2022), Fig. 4, a transformer. Convolutional version: SparK (Tian et al. 2023), ConvNeXt at 60%.
- The paper calls these plausible; the pepper is our reading
- How each builds its target, and how each avoids constant outputs
- Transformer numbers in the Excels row. Protocols differ across papers
- DINOv2 and v3 combine self-distillation with masked prediction
- ImageNet linear readout on frozen features, each point the best method of its year as reported in the paper, largest model
- On the same ResNet-50: DINO and SwAV 75%, SimCLR 69%, supervised 79% (Caron et al. 2021, Table 2)
- Foundation models: pretrained without labels, adapted with a small readout
- The 87% is not plotted: separate evaluation on the same models, from the chapter
- Conwell et al. (2024): training images mattered more than architecture or objective
- Project seed: fix the architecture, vary the objective (labels, DINO, image–text). Which predicts TVSD or Triple-N better, at which block? The three ResNet-50s from the assignment are a start
- These models learn without labels. How do they compare with biological vision?
- These are training procedures, not mechanisms: nobody proposes that cortex masks 75% of an image or keeps a slow copy of itself
- The next slides compare representations, not algorithms
- Correlational by construction: models differ at once in architecture, objective, data and scale
- TVSD is the exception, with continued improvement; the reason for the difference between datasets is open
- More accurate models may use different visual strategies: local texture where people use parts or 3-D structure
- Toward more ecologically valid data: a continuous stream of experience
- Left to right: each step changes what the network must predict. Stream frames are a sliding crop, not video
- A bit of a hack: nobody cares which five minutes a frame came from. It works because objects and scenes change slowly
- MobileNetV2 from scratch, SAYCam (Sullivan et al. 2021), one model per child. Reported model also used color augmentation
- This classifies segments; it is not contrastive. Their contrastive version, neighbors as positives, did clearly worse
- 288 s segments beat 18 s
- Readout on that child's 26 categories: about 60% against 54%. New toy exemplars: about 45% against 60%. Models trained on the other two children, tested on this child's labeled frames, do not beat ImageNet. Plots in the paper; it is on the research trail
- Neural law: a relation between a controlled change in experience and a change in neural responses or perception
- Position tolerance: a neuron keeps its preference across positions
- Li and DiCarlo's monkeys were adults: plasticity, not development; saccade signals not ruled out
- Networks on chicks' views reproduce the viewpoint pattern (Pandey et al. 2024); not yet tested on altered experience
- Garimella et al. (2024): virtual rearing chambers matching the chick experiments (Wood and Wood 2021). Models pretrained at scale can learn to separate objects, so this is about what the stream asks for
- Ashok et al. (2026), from our lab: a dorsal-like and a ventral-like pathway. Infants also see objects first through common motion (Kellman and Spelke 1983)
- The minute paper still counts. Further reading and "Going further" are optional
- Definitions to know for the exam: autoencoder, encoder, decoder, bottleneck, reconstruction loss, sparse code and L1 regularization, generative model, pretext task, linear readout and fine-tuning, positive pair and negatives, collapse, masked prediction, and what the VAE adds to the autoencoder
- The exam covers the main slides of this deck; the further-reading papers are optional
- Before posting, skim the existing entries on your paper; your post must add something not already said
- Submitted on Canvas (Minute papers → Minute Paper 15); credit for a thoughtful attempt
- Optional: the VAE loss; the InfoNCE loss, in an equation and in PyTorch; how much visual experience models and children get; three learning signals in a stream; models that learn from video
- With β = 1, the expected loss over sampled codes is the negative of a lower bound on log p(x), the ELBO, up to a constant. That bound is what variational inference optimizes
- σₓ²: the assumed pixel noise variance, 0.01 for our face network. ⊙: elementwise multiplication
- The KL line is the closed form for a Gaussian code with diagonal covariance against a standard Gaussian
- z⁽ⁱ⁾: view i's vector from the projection head. This z differs from the VAE's sampled code. The code on the next slide averages this loss over all views. The denominator holds the other view and all negatives
- Cosine similarity: dot product over lengths, so only the angle counts
- τ scales similarities before the softmax. Small τ: the most similar wrong view dominates
- van den Oord et al. (2018); SimCLR's form is NT-Xent
- The same cross-entropy you wrote for the classifier in the assignment, applied to a similarity matrix instead of class scores
- With a real encoder in front of it, this loss and a few hundred lines of data augmentation are SimCLR
- Counts depend on what a visual sample is: waking time at 1 to 10 samples per second. Successive views are correlated
- Rough numbers, for scale only
- Crops stand in for the views an animal gets by moving
- Getting a stream does not mean using these signals: many action-recognition categories can be told from a single frame (Sevilla-Lara et al. 2021)
- Predicting subsequent views against interpolation: Zhou et al. (2026), from our lab. Whether generative video models understand physics is an open question