PCA & eigendecomposition

Slide 1

CPSY 1291 Computational Methods for Mind, Brain & Behavior

PCA & eigendecomposition

Tuesday, September 22 · Fall 2026

Return to contents

Slide 2

Last time: comparing representational spaces

• RDMs describe which stimuli a system treats as similar or different.
• Comparing RDMs lets us ask: do two systems organize the same stimuli similarly?
• MDS recovers a psychological space whose distances approximate the judged dissimilarities.

Return to contents

Slide 3

How many dimensions describe the variation?

• MDS summarized dissimilarities as coordinates in a psychological space. Now return to the response matrix: hundreds of measured dimensions—pixels, features or neurons—for each stimulus.
• If many responses vary together, their variation may occupy fewer dimensions: a smaller embedding dimension, the number of flat (linear) directions that hold most of it. How many are needed, and what is lost when we retain only some of them?

Return to contents

Slide 4

Can fewer coordinates capture the variation?

Which directions capture most of the variation, and what do we lose when we retain only a lower-dimensional description?

• Start with two pixels from the same photographs.
• Find a direction, project onto it, and reconstruct.
• Use the same operations to study images and neural responses.

Return to contents

Slide 6

Why look for a direction at all?

• Along the long arrow, both pixels brighten or darken together. Its weights are close to $(1,1)/\sqrt{2}$, so it behaves like their average and accounts for 90% of the variation.
• Along the short arrow, one pixel brightens as the other darkens. Its weights are close to $(1,-1)/\sqrt{2}$, so it compares the two pixels and accounts for the remaining 10%.
• For this pixel pair, the two fitted directions are close to average and difference.

Return to contents

Slide 7

From the same dots to one distribution

• These are the same 120 dots. For the moment, read each dot's horizontal $x_1$ coordinate and set $x_2$ aside.
• The histogram is the distribution of those 120 values. Their mean is 101.6; their spread tells us how much this measurement varies across photographs.

Return to contents

Slide 8

From observed values to a Gaussian model

• The previous histogram showed observed pixel values. This is an idealized reference model, not an assumption today's method requires.
• A Gaussian is smooth and bell-shaped. Its mean $\mu$ gives the center and its standard deviation $\sigma$ gives the spread; about 68% lies between $\mu-\sigma$ and $\mu+\sigma$.

Return to contents

Slide 9

The Gaussian model in two dimensions

• Each simulated point contains two measurements, $(x_1,x_2)$. These are draws from the model, not the 120 photographs.
• The mean becomes a mean vector $\boldsymbol{\mu}$, the center of the points; the covariance matrix describes their width, elongation and tilt.
• PCA will find the directions along which the ellipse is longest and shortest.

Return to contents

Slide 11

Where are the points, and how do they spread?

• So subtract the mean from every point: $\tilde{\mathbf{x}}^{(i)} = \mathbf{x}^{(i)} - \boldsymbol{\mu}$. This is centering, and it slides the points until their centre sits at the origin.
• PCA describes variation around the mean.
• A negative centered value means “darker than this pixel’s average,” not a negative raw intensity.

Return to contents

Slide 12

Centering, in code

• features is the response matrix from the first lectures: one row per stimulus (images here), one column per coordinate.
• The mean is a vector with one entry per coordinate. Subtracting it from every row leaves each column summing to zero.
• Keep the mean. PCA works on differences from it, and reconstruction adds it back.

Return to contents

Slide 13

Covariance: do the deviations go together?

• These are the same centered responses: $\tilde{x}_1=x_1-\mu_1$ and $\tilde{x}_2=x_2-\mu_2$. Same signs give a positive product; opposite signs give a negative product.
• Covariance is their average product:
  $\displaystyle \operatorname{cov}(\tilde{x}_1,\tilde{x}_2)=\frac{1}{N}\sum_i\tilde{x}_1^{(i)}\tilde{x}_2^{(i)}$.
• Pearson correlation is covariance measured in standard-deviation units:
  $\displaystyle r=\frac{\operatorname{cov}(\tilde{x}_1,\tilde{x}_2)}{\sigma_1\sigma_2}$.

Return to contents

Slide 14

The covariance matrix

• In $\mathbf{C}$, the diagonal contains the variances. The same covariance appears on both sides of the diagonal because $\operatorname{cov}(x_1,x_2)=\operatorname{cov}(x_2,x_1)$.
• Geometrically $\mathbf{C}$ summarizes the points’ spread and orientation: round, an ellipse, or an ellipse tilted because the two vary together.
• The ellipse's long direction is its direction of greatest variance.

Return to contents

Slide 17

Projection onto a unit direction

• The foot of that perpendicular is $\mathbf{p}=(\mathbf{w}^\top\tilde{\mathbf{x}})\,\mathbf{w} = z\,\mathbf{w}$: the part of $\tilde{\mathbf{x}}$ that lies along $\mathbf{w}$.
• Here $K=1$: two coordinates become one score, $z$.
• More generally, $K$ retained directions replace $D$ coordinates with $K$ scores.

Return to contents

Slide 18

A direction keeps part of the spread

• Project every point onto some direction. The red segments are what is left over: the part each point keeps to itself.
• At this angle the direction keeps 23% of the total spread and leaves 77% over.

Return to contents

Slide 19

Turn it until the leftovers are smallest

• Turn the direction until those segments are as short as possible: here at 44°, lengthwise through the points.
• Now it keeps 90% and leaves 10%.

Return to contents

Slide 20

Spin the direction yourself

• Before turning the direction, predict which direction leaves the smallest error.
• Projected variance and squared reconstruction error add to a fixed total. Maximizing one minimizes the other.

Drag a unit direction through the centered points of the 120 animal photographs and watch the projected variance and the reconstruction error trade off; their sum is fixed — open interactive version

Return to contents

Slide 21

From turning the line to a calculation

• For a unit direction $\mathbf{w}$, each centered response has one score: $z=\mathbf{w}^\top\tilde{\mathbf{x}}$.
• Across all stimuli, those scores have variance

$$\operatorname{Var}(z)=\mathbf{w}^\top\mathbf{C}\mathbf{w}.$$

• The best direction is the $\mathbf{w}$ that makes this variance largest. The covariance matrix $\mathbf{C}$ lets us find it without searching every angle.

Return to contents

Slide 22

The covariance matrix changes a direction

• The blue vector $\mathbf{w}$ is one possible direction through the response space.
• Multiply it by the covariance matrix: $\mathbf{C}\mathbf{w}$ is the red vector.
• Usually, the red vector has a different length and points in a different direction.

Return to contents

Slide 23

Most directions turn

• Apply $\mathbf{C}$ to every direction on the unit circle. The transformed tips trace an ellipse.
• Almost every direction comes out turned as well as rescaled.

Return to contents

Slide 24

Eigenvectors are directions that do not turn

• For most vectors, $\mathbf{Cw}$ points in a different direction from $\mathbf{w}$.
• An eigenvector $\mathbf{v}_k$ is special: $\mathbf{C}$ changes its length but leaves it on the same line.
• The amount of scaling is its eigenvalue $\lambda_k$:

$$\mathbf{C}\mathbf{v}_k=\lambda_k\mathbf{v}_k.$$

Return to contents

Slide 25

An eigenvalue gives variance along its eigenvector

• Put $\mathbf{w}=\mathbf{v}_k$ into the variance formula, then replace $\mathbf{C}\mathbf{v}_k$ by $\lambda_k\mathbf{v}_k$:

$$\operatorname{Var}(z_k)=\mathbf{v}_k^\top\mathbf{C}\mathbf{v}_k=\mathbf{v}_k^\top(\lambda_k\mathbf{v}_k)=\lambda_k\,\mathbf{v}_k^\top\mathbf{v}_k=\lambda_k,$$

since $\mathbf{v}_k$ has length 1.

• The largest eigenvalue identifies the direction with the most variance. Here it is the same 44° direction found by turning the line.
• Ranked by eigenvalue, these directions are the principal components. The first is PC1; computing them is principal component analysis (PCA).

Return to contents

Slide 26

PCA = eigendecomposition of $\mathbf{C}$

Covariance (centered, $N$ samples): $\;\mathbf{C} = \dfrac{1}{N}\sum_{i} \tilde{\mathbf{x}}^{(i)}\tilde{\mathbf{x}}^{(i)\top}$   (code divides by $N-1$; same eigenvectors)

Eigendecomposition (orthonormal $\textcolor{#B5396B}{\mathbf{V}}$):

• Principal directions = eigenvectors $\textcolor{#B5396B}{\mathbf{v}_k}$ of $\mathbf{C}$.
• Variance along $\textcolor{#B5396B}{\mathbf{v}_k}$ = $\textcolor{#3F7A6B}{\lambda_k}$.
• $\text{PC1} = \textcolor{#B5396B}{\mathbf{v}_1}$, the eigenvector with the largest eigenvalue.

Return to contents

Slide 27

From a line to a plane

• In this picture, $D=3$ measured coordinates and we retain $K=2$ perpendicular directions.
• The two directions define a plane; each centered point gets two scores, $(z_1,z_2)$.
• Keeping all $D$ directions changes coordinates without losing information.
• Keeping $K<D$ projects each point onto a lower-dimensional flat (linear) subspace: a line, a plane, or its analogue in more dimensions.

Return to contents

Slide 28

The new coordinates are the PC scores

For one stimulus, take its dot product with each retained direction:

$$z_k=\mathbf{v}_k^\top(\mathbf{x}-\boldsymbol{\mu}).$$

• $\mathbf{v}_k$ gives weights on the original measurements.
• The score $z_k$ is this stimulus's coordinate along that direction.
• Keeping $K$ directions gives the point $K$ scores.

Return to contents

Slide 29

The same points, in the new coordinates

• Left: points in the original measured coordinates $\tilde{x}_1, \tilde{x}_2, \tilde{x}_3$. Right: the same points at their scores $(z_1, z_2)$: the PCA embedding.
• The green point is the same stimulus in both. Its distance from the plane, the dashed red residual, is what the 2-D map leaves out.

Return to contents

Slide 30

Reconstruction: the way back

• Return to the original measurement coordinates and start from the mean.
• Add each retained direction, weighted by the image's score:

$$\hat{\mathbf{x}}=\boldsymbol{\mu}+\sum_{k=1}^{K}z_k\mathbf{v}_k.$$

• The residual is the difference between the original and reconstructed measurements.

Return to contents

Slide 31

What do we lose when we keep fewer components?

• PCA on the pixels of 400 faces: keeping more components adds detail and reduces reconstruction error.
• A residual fraction of 0 means exact reconstruction; larger values mean more of that face was left out.
• Dots: this face. Curve: the discarded variance across all 400 faces, which need not equal one face's error.

Return to contents

Slide 32

How many components? The scree plot

• These are the same 400 faces. The left panel shows the variance fraction carried by each PC.
• The cumulative curve answers how many PCs are needed to retain a chosen fraction, such as 90%: the embedding dimension at that threshold.
• A variance threshold sets a reconstruction budget. It does not establish performance on a task.

Return to contents

Slide 33

PCA in scikit-learn

• Score: a stimulus's coordinate along a principal direction.
• fit_transform centers each feature across stimuli. It does not z-score them (divide each by its standard deviation).
• SVD computes the PCA directions and variances from the centered data without explicitly forming its covariance matrix.

Return to contents

Slide 34

Can these directions help explain a neuron?

• Inferotemporal (IT) cortex is part of the primate visual pathway involved in object recognition.
• Each matrix shows one cell's responses: 51 objects × 24 views. A bright column means strong responses across views of one object.
• Can a space derived from a network's responses help organize these preferences?

Return to contents

Slide 35

Build an object space from network responses

• Present the 1,224 images to AlexNet and collect responses from fc6, a late network layer.
• Run PCA on those response vectors. In the paper, 50 components account for 85% of the variance.
• The picture is a schematic of the first two PCs, arranged for legibility rather than a raw scatter plot.
• “Stubby–spiky” and “animate–inanimate” are proposed interpretations of those directions. They need testing.

Return to contents

Slide 36

Test whether a direction predicts a response

• The gold ring follows one image from its position in model space to its measured response.
• Choose a direction using training images, then ask whether each image's projection predicts the cell's response on held-out images, which were not used to choose it.
• The preferred direction can combine several PCs; it need not coincide with PC1 or PC2.

Return to contents

Slide 37

Neural preferences occupy different regions

• Grey circles are stimuli in the network-derived PC plane.
• Colored points are images that strongly activate four IT systems. NML is the network Bao et al. found in IT's "no man's land", cortex that no earlier localizer had assigned a category; the numbers in parentheses count recorded neurons.
• Different IT systems prefer different regions of the model space. A separate analysis tests whether a neuron's response varies with its projection along a direction.

Return to contents

Slide 38

Use the space to make a new prediction

• The object space predicted a system responding to stubby, inanimate objects.
• fMRI located it (magenta); targeted recordings confirmed the object preferences.
• A description of a representation can guide a new experiment.

Return to contents

Slide 39

The measurement scale changes PCA

• PCA maximizes variance, so a feature measured in millimetres dominates one measured in metres: in raw units PC1 is $1.00\,x_1 + 0.00\,x_2$ and the second feature is invisible.
• After z-scoring each feature (subtract its mean, divide by its standard deviation), PC1 is $0.71\,x_1 + 0.71\,x_2$ and recovers the diagonal structure the two features share.
• Standardize when the units are not comparable. When they are comparable and the variances differ for real reasons, standardizing changes the relative weighting of features; it is a modelling decision, not a default.

Return to contents

Slide 40

Most variance need not mean most useful

• The largest direction in the 120 photographs carries 29% of the variation and is strongly related to overall brightness: ordered along it they run dark to bright, different categories interspersed.
• Would this direction help distinguish the animals? This plot alone cannot answer.
• PCA ranks directions by variance; the task determines what is useful.

Return to contents

Slide 41

Which direction would you keep?

• The groups vary mainly from left to right. Their labels differ from top to bottom.
• Which direction would best reconstruct the points?
• Which direction would best distinguish the groups?

Return to contents

Slide 42

Low variance can still separate the groups

• PC1 retains 97.8% of variance, but both groups have the same PC1 coordinates.
• PC2 retains 2.2%, yet a threshold at zero separates the groups.
• Variance measures spread. Task relevance depends on what we want to distinguish.

Return to contents

Slide 43

What PCA provides

• PCA replaces many measurements with ordered scores and tells us how much variance each direction retains.
• Those scores support reconstruction and can connect a model representation to neural responses. In the Bao study, that connection helped motivate a new experiment.
• PCA describes variation. A separate test establishes what the variation is useful for.

Return to contents

Slide 44

What if the variation follows a curve?

• Changing one stimulus parameter can trace a curved path in a large response space.
• Even one varying parameter can require two linear coordinates when the path bends back on itself.
• PCA approximates data with a flat subspace.
• Next: nonlinear maps, and how to check what their geometry preserves.

Return to contents

Slide 45

Further reading for the research trail

Handout: Statistics and the mathematics of PCA — variance, covariance, eigenvectors, projection and reconstruction, worked by hand in this lecture’s notation.

• Bao et al. (2020): object-space directions and IT responses. Nature
• Chang & Tsao (2017): face-space directions and individual neurons. Cell
• Turk & Pentland (1991): eigenfaces. J. Cogn. Neurosci.
• DiCarlo & Cox (2007): untangling object representations. Trends Cogn. Sci.
• Stringer et al. (2019): a structured, high-dimensional neural spectrum. Nature

Trail entry: pick any paper cited in a lecture so far, read it, and post four to six sentences on Ed under Research trail — what it did, and what surprised you or what question it left you with. Reply once to a classmate. Next deadline Fri 9/25, best three of four entries count.

Return to contents

Slide 46

Minute paper

Submit on Canvas → Minute papers → Minute Paper 4.

Write three brief points in your own words:

1. Why do we center the data before finding the principal directions?
2. Something you do not yet understand, or a question still open
3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents

Slide 47

Appendix: examples and implementation details

Additional examples, the PCA/SVD implementation, and reference material for the assignments.

Return to contents

Slide 48

One direction was not enough

• Fit the same ratings on one direction and the categories pile up: the distances in the fit no longer match the judged dissimilarities.
• A second direction removes most of that distortion. A third and a fourth remove very little. Two directions greatly improve the fit, but some distortion remains.

Return to contents

Slide 49

How far does the dependence reach?

• The same pixel-pair computation, at larger separations: one pixel against another, across the same 120 photographs.
• Next door, the two pixel values strongly covary. Twenty pixels apart, they barely covary.
• Neighbouring measurements are strongly dependent, and the dependence is local.

Return to contents

Slide 50

PCA in code: two routes to the same answer

• Use eigh, not eig: $\mathbf{C}$ is symmetric, so the eigenvalues are real and eigh is faster and stabler. It returns them ascending, so the reversal is not optional.
• Prefer route 2. Forming $\mathbf{C}$ costs $D\times D$ memory; for a $256\times256$ color image $D\approx196{,}608$ and $\mathbf{C}$ would be ~309 GB in float64. SVD never builds it. Sign is arbitrary: $\mathbf{v}_k$ and $-\mathbf{v}_k$ are both valid; Assignment 2's helper fixes it so the largest entry is positive.

Return to contents

Slide 51

Eigenfaces

PCA on 400 faces, and a classic approach to face recognition.

Return to contents

Slide 52

PCA as compression: eigenfaces

Stack face images as vectors and run PCA → the leading eigenvectors are eigenfaces: ghostly, face-like basis images, one direction in pixel space each.

Return to contents

Slide 53

Face space

• 400 face images ($64\times64$ pixels, so $D = 4096$), PCA on the centered pixels, and twelve of the faces drawn at their coordinates $(z_1, z_2)$ on the first two eigenvectors.
• Two numbers per face instead of 4,096. They carry 24% and 14% of the variance: a coarse, lossy summary, but already one in which lighting and head pose vary smoothly across the plane.
• This is a face space: every face is a point, and nearby points are faces that look alike to the two leading eigenfaces. The preceding slide shows those eigenfaces as images.

Return to contents

Slide 54

A face = mean + a few eigenfaces

• These faces can be approximated by the mean face plus a weighted sum of eigenfaces. More components add finer detail.
• Turk & Pentland used the coefficients for recognition: a new face is matched to the stored face with the nearest coefficients.

Return to contents

Slide 55

Mix your own face

• Each slider sets one coefficient in mean + weighted eigenfaces.
• Set a coefficient to zero to remove that direction’s contribution.

Eigenface mixer: sliders on the first eight eigenface coefficients rebuild a face from the mean; a second tab reconstructs a stored face with a growing number of components; a third lets you click anywhere on the PC1–PC2 plane to see the face there — open interactive version

Return to contents

Slide 56

Two views, one answer

• Maximize the variance of the projections: $\displaystyle\max_{\lVert\mathbf{v}\rVert=1}\mathbf{v}^\top\mathbf{C}\,\mathbf{v}$
• Minimize the squared reconstruction error: $\displaystyle\min_{\lVert\mathbf{v}\rVert=1}\tfrac{1}{N}\sum_i\lVert\tilde{\mathbf{x}}^{(i)} - (\mathbf{v}^\top\tilde{\mathbf{x}}^{(i)})\mathbf{v}\rVert^2$
• These are the same problem: total variance is fixed, so whatever the projection captures, the residual loses. Both are solved by the top eigenvector of $\mathbf{C}$.
• For $K$ retained PCs, the average squared fitting-data reconstruction error is $\sum_{k>K}\lambda_k$ (using covariance divided by $N$).
• Connection to MDS: classical MDS on a Euclidean dissimilarity matrix returns the PCA coordinates up to an orthogonal transformation of the centered data.

Return to contents

Slide 57

PCA as decorrelation & whitening

• Project onto the eigenbasis and the new coordinates are uncorrelated: the covariance becomes diagonal. For positive eigenvalues, rescale each retained direction by $1/\sqrt{\lambda_i}$ and you get whitening: zero correlation, unit variance.

• Why analysts reach for it:
  • remove redundancy between measured dimensions before modeling
  • decorrelate inputs, without assuming the resulting features are independent
  • compress by dropping directions; whether this denoises depends on what they contain

Return to contents

Slide 58

What do the components look like?

• Each eigenvector is a vector in the original space, so each PC unrolls into a picture. Here the inputs are synthetic patterns: early components recover structured variation; later ones contain little variance.
• The components can be read as weights on the original variables, then tested for what they capture.

Return to contents

Slide 59

The rainbow cortex is three numbers per voxel

• Over two hours of narrated stories, one regression per voxel: every voxel gets a vector of weights on word-related features, and cortex is one more $N\times D$ matrix.
• PCA on it, keep three components, color each voxel by its coordinates: PC1 → red, PC2 → green, PC3 → blue. The palette is the projection $\mathbf{z}$.

Return to contents

Slide 60

PCA finds flat subspaces

• If the data lie on a curved manifold, a smooth curved surface (a spiral, a Swiss roll, a torus), a low-dimensional linear projection can fold distinct regions onto each other.
• Two points far apart along the manifold can land on top of each other after projection: the two arms are perfectly distinct in the plane and inseparable along PC1.
• t-SNE and UMAP construct nonlinear maps emphasizing local relationships. Their displays can distort global structure and need their own checks.

Return to contents