Nonlinear dimensionality reduction

Slide 1

CPSY 1291 Computational Methods for Mind, Brain & Behavior

Nonlinear dimensionality reduction

Thursday, September 24 · Fall 2026

Return to contents

Slide 2

Your questions from Tuesday

• "If there are 50 principle components, why is it/how can it be represented by just 2 axes as on the ppt slides?": it cannot. A PCA plot keeps the two directions along which the points vary most over the whole data set and drops the other 48. Points that differ only along the dropped directions land on top of each other. That is where today's nonlinear methods come in: instead of keeping a few directions, they keep relationships between points, above all who is whose neighbor.
• "Does this mean neural signals are operating in 2 dimensional space": no. For the Bao et al. objects in AlexNet fc6 (Assignment 1), the first two components hold only 22% of the variance, and 85% takes 87 (Bao et al. report about 50). Counting components to a threshold measures the embedding dimension, a linear count, and treats the rest as noise. If the data lie on a curved surface, the intrinsic dimension, the number of dimensions needed to locate a point on it, can be far smaller.

Next: what every 2-D picture of the data gives up.

Return to contents

Slide 4

Every low-dimensional picture is an approximation

• PCA keeps each point's projection onto the plane of largest variance; MDS keeps the distances between points.
• PCA drops each point's distance from the plane, the dashed red residual, assuming it is noise.
• MDS cannot drop it: to keep every distance, it spreads the residual into the map, which distorts it.

Green point, far above the plane: PCA drops it among the others; MDS pushes it to the edge.

Return to contents

Slide 5

Real images curve: turning one photograph

• Why look? Recognition needs one object's images to stay together. In pixels, turning alone sends the fish on a loop that needs 24 PCs for 90%, yet one dimension, the angle, locates every image.

Return to contents

Slide 6

Shifting traces a curve too

• Slide the photograph left to right: one knob, one curve. It bends, so it is not a straight line in pixel space.

Return to contents

Slide 7

Enlarging traces another curve

• Each knob gives a one-dimensional curve. Turn all three at once and the images fill a three-dimensional curved surface.

Return to contents

Slide 8

All three changes in one PCA

In pixels, one fish's images spread over many curved directions. Does a trained network pull them together?

Return to contents

Slide 9

Turn the knobs yourself

Pick one of seven objects and one change, turn, shift or enlarge, and drag the slider: the current image is shown large beside the curve that all images of that change trace on their first two principal components — open interactive version

Return to contents

Slide 10

What is a manifold?

• Turn and enlarge one photograph: red rings are one size turned round; green spokes are one angle enlarged.
• The images fill a curved surface: a manifold. Up close (the gold cell) it is flat.
• Intrinsic dimension: how many dimensions are needed to locate a point on it? Here 2, the angle and the size.
• Ambient space: where it sits, 9,216 pixel values.

Next: many objects, each with its own manifold, in a trained network.

Return to contents

Slide 11

Objects seen from many viewpoints

• 1,224 images: 51 objects, each rendered from 24 views, in 6 categories.
• Bao, She, McGill & Tsao showed these images to monkeys while recording from IT cortex, the late stage of the visual system where neurons respond to whole objects. You also use them in Assignment 1.
• Here each image is a point in the late layer of ResNet-18, the trained image network of Assignment 1: 4,096 numbers per image.
• Two levels of grouping: views of the same object, and objects of the same category.
• The question: pixels scatter an object's views. Does the network keep them together, and keep different objects apart?

Return to contents

Slide 12

One object, 24 views: one patch

• Each dot is one photograph of the same excavator, placed by the network's response to it. The axes are the two principal directions of those 24 responses.
• Very different images, one patch: all the ways this object can look.
• Compare the pixels, where turning alone traced a long loop: the network gathers the views.

Return to contents

Slide 13

Three objects, three patches

• Now one PCA of all 72 responses together. Each object keeps to its own small patch, and the patches are far apart.
• Viewpoint moves a response within a patch; identity moves it between patches.
• And the brain? Bao et al. recorded monkey IT cortex with these same images: do its neurons also keep each object together? That question drives the recognition lectures.

Return to contents

Slide 14

Drawn in 2007, computed today

One sheet per object. In a trained network, the sheets of different objects sit apart. In pixels, they do not: next slide.

Return to contents

Slide 15

Two sheets, tangled

• In pixel coordinates the sheets of two people interleave.
• Recognizing who it is means finding a flat (linear) boundary, a hyperplane, with one person's sheet on each side. Here none exists.
• That boundary is what a linear classifier learns. A good representation untangles the sheets; testing that with a linear readout is the next lecture.

Next: to see whether a representation untangles them, we need to draw manifolds.

Return to contents

Slide 16

Untangled: a flat boundary now works

• Same two objects, two representations: in pixels the sheets are linked; in a later layer they are pulled apart
• Untangling means a flat boundary can now separate the objects

Return to contents

Slide 17

How do we see a manifold?

To ask whether object manifolds (the responses to one object across all its views) are tangled or untangled, we have to look at them, but they live in thousands of dimensions. We need a 2-D map that keeps their shape. First, test the maps on a manifold whose true shape we know.

Return to contents

Slide 18

Test case: can PCA unroll a known manifold?

• 1 → 2: a 2-D sheet rolled up in 3-D. A and B are 11.5 apart in a straight line (Euclidean distance in 3-D), but 59.5 apart walking along the sheet.
• 3: PCA is linear: it can only rotate the data and project onto a flat plane. It keeps 71% of the variance, yet folds the roll: A and B land on the same spot.

Return to contents

Slide 19

Linear methods fold the roll

• PCA keeps the directions of largest variance; MDS keeps the straight-line distances. Both see the data through a flat lens.
• Points on neighboring turns are close through the air, so both methods put them close. Neither recovers position along the sheet. The fix: better distances, not a better projection.

Return to contents

Slide 20

Perich's origami crane

• A crane is folded from one flat square: nothing is cut or glued.
• It sits in 3-D, yet it is still a 2-D sheet: unfold it and you get the square back.
• If the square had text printed on it, the crane would hide it. To read it, you must unfold, not just look from a better angle.

Return to contents

Slide 21

Squashing is not unfolding

• PCA picks a flat view, so the panels land on top of each other. A nonlinear method aims to unfold the sheet.

Return to contents

Slide 22

Three ways to count dimensions

• The three numbers answer different questions; none is the dimension.
• PCA measures the embedding dimension, a linear count that needs a variance threshold (90% here). It overestimates the intrinsic one whenever the manifold curves.
• For real data such as the network's responses to the Bao objects, the intrinsic dimension is unknown: that is what the nonlinear methods coming next try to reveal.

Return to contents

Slide 23

How could we fix this?

Keep the MDS idea, but change the distances. Instead of measuring straight through the air, measure along the data: only step from a point to its nearest neighbors, and add up the steps.

Return to contents

Slide 24

Isomap: measure "along the sheet", then run MDS

• Connect each point to its 10 nearest neighbors. The shortest path through that graph approximates the "along-the-sheet" distance: 71.2 for the two probes, 6.1 in a straight line.
• Run MDS, from the second lecture, on those path lengths. The roll unrolls if the points cover the sheet densely and no edge jumps between turns.

Return to contents

Slide 25

Folded paper: you cannot read it flat

• The crane, computed: a page printed with text, rolled into a Swiss roll.
• PCA gives the best flat view, but the letters pile up. To read the text, you must unfold the sheet: a nonlinear method such as Isomap (panel 4).

Return to contents

Slide 26

Isomap in practice

• One wrong edge between turns (a short circuit) and the roll folds again.
• It also needs densely sampled data without holes, and becomes slow for large data sets.
• So Isomap is rarely used today. The popular modern methods are t-SNE and UMAP; we focus on t-SNE, which you use in Assignment 1.

Return to contents

Slide 27

t-SNE

Convert distances into neighbor probabilities, then find a low-dimensional layout, usually 2-D, that approximates those neighbor probabilities.

Return to contents

Slide 28

What t-SNE is trying to do

• Like MDS, t-SNE places the points in a low-dimensional map, usually 2-D (here 1-D, a line, so the example is easy to see), so that relationships in the map match relationships in the data.
• MDS tries to match all distances. t-SNE matches neighbors: close in the data stays close in the map; large distances are not kept.
• t-SNE does not look for clusters: clusters survive because their members are each other's neighbors.
• Three steps: (1) data: distances → neighbor probabilities $P$; (2) map: similarities $Q$; (3) move the map points until $Q$ matches $P$.

Return to contents

Slide 29

Step 1 — in the data: distances → $P$

• Center a Gaussian on each point $\mathbf{x}^{(i)}$ and normalize:

$$p_{j\mid i}\;\propto\;\exp\!\left(-\frac{\lVert\mathbf{x}^{(i)}-\mathbf{x}^{(j)}\rVert^{2}}{2\sigma_i^{2}}\right)$$

• Read it as: "if I am point $i$, how likely am I to pick $j$ as my neighbor?" In MDS terms, $P$ replaces the dissimilarities, warped: near pairs count a lot, far ones almost nothing.
• Each point gets its own width $\sigma_i$, a free parameter: narrow in dense regions, wide in sparse ones.
• Make it symmetric: $p_{ij}=\dfrac{p_{j\mid i}+p_{i\mid j}}{2N}$, one probability per pair.

Return to contents

Slide 30

Perplexity: how many neighbors each point listens to

• You do not set each point's width $\sigma_i$ by hand. You choose one number, the perplexity: roughly, how many neighbors each point should pay attention to.
• t-SNE then widens or narrows each point's Gaussian until it has about that many neighbors: narrow in dense regions, wide in sparse ones.

Return to contents

Slide 32

Perplexity changes the map

• Two clusters of 50 points each, 100 in all. Perplexity is a number of neighbors, so it must stay below the number of points: the last panel, perplexity 100, is not valid.
• Typical range 5–50. Run a sweep and trust only what holds across it.

Stable across 5–50: two groups that never mix. That, and only that, is worth interpreting.

Return to contents

Slide 33

Step 2 — in the map: similarities $Q$

• In the map (in MDS, the map distances): similarity of map points with a Student-$t$ kernel, not a Gaussian: $\;q_{ij}\;\propto\;\left(1+\lVert\mathbf{y}^{(i)}-\mathbf{y}^{(j)}\rVert^{2}\right)^{-1}$
• Why the t? A point can have many equally-near neighbors in the data, but a 2-D page fits only a few around it (crowding). The heavy tail lets the rest sit farther out.

Return to contents

Slide 34

Step 3 — move the map until $Q$ matches $P$

• As in MDS: drop the points on the line at random, measure how badly $Q$ matches $P$, nudge every point a little to reduce the mismatch, repeat.
• The mismatch is the KL divergence, $\sum_{i\neq j} p_{ij}\log\frac{p_{ij}}{q_{ij}}$; the nudging is gradient descent. Both return when we train neural networks.
• The random start is set by a random seed: another seed gives another map. Starting from the PCA map (PCA initialization) is more repeatable.

Return to contents

Slide 35

What the mismatch punishes

• Tearing one point from its neighbors costs more than pushing a whole group of eight against another.
• Close pairs (large $p_{ij}$) dominate the mismatch; far pairs barely count.
• So t-SNE keeps neighbors, not distances. A t-SNE map is not a projection: distances, sizes and gaps in it are not distances in the data.

Return to contents

Slide 36

UMAP: a related way to build a neighborhood map

• Connect each point to nearby points in the original space, giving stronger connections to closer neighbors.
• Lay out the map to favor those connections, while keeping unrelated points from collapsing together.
• n_neighbors sets the neighborhood scale; min_dist how tightly points pack.
• Same data, same lesson: both maps keep local neighbors and tear the sheet in places.

Return to contents

Slide 37

UMAP's two settings

• n_neighbors plays the role of perplexity: how many neighbors each point listens to. Few: many small pieces. Many: more global shape.
• min_dist sets how tightly points may pack: 0 gives thin, tight clumps; larger values spread them out. It changes the look, not the neighbors.
• Same rule as t-SNE: sweep, and trust what holds across settings.

Return to contents

Slide 38

Match the method to the relationship

Each map keeps only what its method keeps. Before trusting a pattern, check that the method keeps that kind of relationship.

Return to contents

Slide 39

Which one should I use?

• Start with PCA: linear, fast, applies to new data, and tells you how much variance the 2-D view keeps.
• To look at neighborhoods and clusters, use t-SNE or UMAP. They are close cousins; Assignment 1 uses t-SNE only to keep it short, and UMAP would teach the same lessons.
• Whichever you use: PCA initialization, sweep perplexity (t-SNE) or n_neighbors (UMAP) and the random seed, and keep what holds across the sweep.
• Never pick the prettiest run. If a map shows two groups, check that the images really group that way in the network's own responses, not just in the picture.

Next: back to our own data. Which features of a map can you believe?

Return to contents

Slide 40

Back to our data: same responses, two maps

Suppose we plot the same network responses twice: in the PCA map the objects overlap, in the t-SNE map they form separate islands.

• Did the network's responses change? No: only the plotting method did.
• Did the PCA map hide groups that are really there? Possibly: groups that differ only along the dropped directions overlap in two PCs.
• Or did t-SNE exaggerate the groups? Possibly too: it can make clumps, and its gaps are not distances.
• Only the original responses can tell: check each image's nearest neighbors there, or train a classifier on some images and test it on others.

A map that looks cleaner does not mean the responses carry more information.

Return to contents

Slide 41

The Bao objects, four ways — live

Interactive map of the network's responses to the Bao objects (51 objects, 8 views each): t-SNE running live in the browser, with perplexity, steps and seed controls, next to precomputed UMAP, PCA and MDS layouts of the same points; hovering over an image shows it large and draws lines to its 10 nearest neighbors in the original responses; meters give neighbor agreement and object purity — open interactive version

Return to contents

Slide 42

Which claims survive a change of map?

Wattenberg, Viégas & Johnson (2016) built toy data with known structure and swept every setting. Their examples show what a visualization can change.

Return to contents

Slide 43

What a t-SNE map can fake

Assignment 1 has you test each of these on real network responses: perplexity sweeps, seeds, and what two dimensions lose.

Return to contents

Slide 44

A map can be stable and still distort the data

Suppose two runs give the same ten nearest neighbors for an image, but only four are among its ten nearest neighbors in the original space.

Repeatability, fidelity and object grouping are different questions. Assignment 1 has you compute each one.

Next: if a map cannot settle a question, what can?

Return to contents

Slide 45

From a map to a test of the representation

A map suggests the question; the response space answers it.

Return to contents

Slide 46

What have we learned about a representation?

• t-SNE and UMAP emphasize local neighborhoods; distant pairs are only weakly constrained. Sizes and gaps in the map are not measurements.
• Dimension is ambient, embedding (what PCA sees) or intrinsic (the surface's own). Isomap keeps "along-the-sheet" distances and can recover the intrinsic shape when the neighbor graph stays on the sheet.
• A map can reveal what a two-component PCA projection hides, and it can invent structure. Stability does not establish fidelity.
• To interpret grouping, check neighborhoods and labels in the response space. To interpret task information, test a readout on new data.

A useful map helps us formulate a claim that we can test.

Return to contents

Slide 47

Further reading

• Required reading: Wattenberg, Viégas & Johnson, How to Use t-SNE Effectively (2016)
• Short essay: Perich, Dimensionality: neuroscience's red herring? (2026)
• Research-trail options: Chaudhuri et al., head-direction ring (2019) · Gardner et al., grid-cell torus (2022) · Nieh et al., abstract knowledge in the hippocampus (2021) · Engels et al., circles in language models (2025)
• Optional: Sainburg, Thielk & Gentner, vocal repertoires (2020) · Chari & Pachter, The specious art of single-cell genomics (2023)

The paper you select for your research trail is required; the other research papers are optional.

Return to contents

Slide 48

Minute paper

Submit on Canvas → Minute papers → Minute Paper 5 (access code read out in class).

Write three brief points in your own words:

1. A t-SNE map shows the objects as separate islands. Name one way to check whether those groups are really there in the network's responses.
2. Something you do not yet understand, or a question still open
3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents

Slide 49

Appendix: extra examples

Published examples of each method (PCA, MDS, Isomap, t-SNE, UMAP), two brain examples (a ring and a torus), the Swiss-roll demo with PCA, Isomap and UMAP, a second application of UMAP, and the precise definition of perplexity.

Return to contents

Slide 50

In the wild: PCA

Return to contents

Slide 51

In the wild: MDS

Return to contents

Slide 52

In the wild: Isomap

Return to contents

Slide 53

In the wild: t-SNE

Return to contents

Slide 54

In the wild: UMAP

Return to contents

Slide 55

Sometimes the curved shape is the answer

• Head-direction cells in the mouse anterodorsal thalamus: one point per time bin, one dimension per neuron.
• Isomap closes those points into a ring: one curved dimension, the shape a circular variable should have. A topological test (persistent homology) confirms the loop without trusting the picture.
• Position around the ring tracks the animal's head direction, and the ring persists in REM sleep: a circuit that maintains an internal direction.
• Language models do it too: days of the week and months sit on circles inside GPT-2 and Mistral.

Return to contents

Slide 56

PCA says six; a nonlinear map says torus

• Grid cells are neurons in the rat brain that fire whenever the animal is at the corners of a triangular grid laid over the floor. PCA puts the activity of hundreds of them in about 6 dimensions.
• UMAP shows a torus: an intrinsically 2-D surface, as theory predicted. Persistent homology counts one piece, two loops, one cavity: a torus, tested, not eyeballed.

Return to contents

Slide 57

Unroll the Swiss roll

A rotatable Swiss roll with an unroll slider and two probe points showing straight-line distance against distance along the sheet, beside PCA, Isomap and UMAP embeddings of the same points — open interactive version

Return to contents

Slide 58

Are vocal repertoires discrete or continuous?

• Each point represents a vocal element, described by its spectrogram.
• The maps suggest differences in how strongly vocal elements form groups.
• The paper also measures clusterability in UMAP space. A number makes the description precise, but still depends on the embedding.
• What would support a claim about the animals? Check acoustic relationships, labels or behavior as well.

Return to contents

Slide 59

Perplexity, precisely

• For point $i$, collect its neighbor probabilities from Step 1 into one distribution, $P_i=\{p_{j\mid i}\}_{j\neq i}$ (one row of the conditional probabilities).
• Its entropy, $H(P_i)=-\sum_j p_{j\mid i}\log_2 p_{j\mid i}$, measures how evenly point $i$ spreads its probability: low for a narrow Gaussian, high for a wide one.
• The perplexity is $2^{H(P_i)}$. t-SNE finds, for each point separately, the $\sigma_i$ that makes $2^{H(P_i)}$ equal to the value you chose (a one-dimensional search, by bisection).
• Why "number of neighbors": if point $i$ spread its probability evenly over $k$ neighbors, $H=\log_2 k$ and the perplexity would be exactly $k$.

Return to contents