Geometry, baselines & tests

Slide 1

CPSY 1291 Computational Methods for Mind, Brain & Behavior

Geometry, baselines & tests

Tuesday, October 6 · Fall 2026

Return to contents

Slide 2

Flexible or abstract: can one population be both?

The geometry of a population: how it arranges its conditions decides whether it is flexible (shattering) or abstract (CCGP)

Return to contents

Slide 3

Reminder: shattering measures flexibility

• Flexible: a downstream neuron can learn many groupings of the same stimuli
• Shattering accuracy: fit a readout for each split, score it on held-out trials, and average. 0.5 = guessing
• 3 stimuli in 2 units, in general position: every split is separable, shattering 1.00. On one line, 2 of the 8 splits get at most 2 of 3 right: 0.92
• In general: $N$ stimuli in general position in $D = N - 1$ units, every split is separable
• More dimensions, more splits solved. But it says nothing about stimuli never seen

Return to contents

Slide 4

Reminder: CCGP measures abstraction

• CCGP (cross-condition generalization performance): fit an "animate or not" readout on dogs and chairs, then test it, weights fixed, on cats and tables
• A high CCGP means the code for animacy is abstract: not tied to the training examples (dogs, chairs), so it carries over to new ones (cats, tables)

Return to contents

Slide 5

The task: a hidden context

Schematic of the task of Bernardi et al. (2020)

• Memorized associations are not enough: the monkey must hold context
• An uncued context flips the rules. Neurons are read before the response: action and reward are from the previous trial

Return to contents

Slide 6

The monkeys infer the hidden context

• After one surprise, the monkeys switch every rule
• Accuracy on the last image before a switch, then on the first showing of each image after it
• 1st image after the switch: about 9% correct. Nothing warned the monkey
• 2nd to 4th images, not yet seen in the new context: above chance (dashed line, 50%)
• Bernardi et al. call this inference: it needs a context variable coded apart from any one image
• Which geometry of the conditions allows this? Shattering and CCGP measure it

Return to contents

Slide 7

Toy geometry: clustered by context

Simulated: a hypothetical population of 3 neurons (the axes), one dot per trial, after Bernardi et al. (2020), Fig. 2

• Abstract, not flexible
• Red/blue: context 1/2. Gold plane: the decision boundary of a readout that decodes the trial's context, fitted on A+ and C+ (filled) and tested, weights frozen, on C− and A− (hollow)
• The conditions cluster by context
• CCGP 1.0: context transfers. Shattering 0.73: conditions within a context cannot be told apart. PR 1.01: almost one direction, context

Return to contents

Slide 8

Toy geometry: points in general position

Simulated: a hypothetical population of 3 neurons (the axes), one dot per trial, after Bernardi et al. (2020), Fig. 2

• Flexible, not abstract
• Gold plane: the context readout's decision boundary (filled: fitted on; hollow: never seen)
• $N = 4$ conditions at random in $D = 3$ units: general position, so every split is separable ($D = N - 1$). PR 2.77
• Context can be decoded (shattering 1.0), but CCGP is 0.50: the plane fitted on A+, C+ cuts right through C− and A−

Return to contents

Slide 9

Toy geometry: a square

Simulated: a hypothetical population of 3 neurons (the axes), one dot per trial, after Bernardi et al. (2020), Fig. 2

• Abstract, but not fully flexible
• Gold plane: the context readout's decision boundary (filled: fitted on; hollow: never seen)
• Corners of a square (PR 2.00): one side is context ({A+, C−} vs {A−, C+}), the other reward ({A+, C+} vs {A−, C−})
• CCGP 0.78 (context), 0.72 (reward). Shattering 0.82: the diagonal split, image A (A+, A−) vs image C (C+, C−), fails. It is context XOR reward

Return to contents

Slide 10

Toy geometry: the square, bent

Simulated: a hypothetical population of 3 neurons (the axes), one dot per trial, after Bernardi et al. (2020), Fig. 2

• Flexible and abstract at once
• Gold plane: the context readout's decision boundary (filled: fitted on; hollow: never seen)
• Bending the square adds a dimension: the A and C conditions move apart along a third direction, so the four no longer fit in a plane (PR 2.00 → 2.45)
• Shattering 1.0: the diagonal split now separates. CCGP: 0.78 → 0.77. PR helps shattering, not generalization

Return to contents

Slide 11

Try it: flexible or abstract?

• Mix: cube (0) to general position (1). Shattering rises, CCGP falls, decoding stays high

Eight task conditions (context, action and reward, each two-valued) as responses of 30 model units. A slider mixes the conditions from the corners of a cube (abstract) toward random positions (flexible). A bar chart updates live: shattering accuracy as a black line, CCGP for context, action and reward as filled bars, decoding with all conditions in training as outline bars, chance at 0.5. A map shows the eight condition means, with gold segments joining conditions that differ only in context, and an inset shows how many directions the means use (participation ratio) — open interactive version

Return to contents

Slide 12

Hippocampus: context transfers, action does not

• In all three areas context transfers and shattering is high, like the demo's compromise
• Both scores are average accuracies of linear readouts; chance is 0.5 (dashed)
• Black: CCGP (red context, orange previous action, purple reward, the paper’s ‘value’). Grey: shattering. Dots: the authors' model
• Context CCGP: 0.96 HPC, 0.74 DLPFC, 0.81 ACC. Shattering: 0.70, 0.75, 0.74
• Previous action in HPC: 0.44: below the dashed line, but within chance range (0.41–0.59). Not abstract, yet decodable

Return to contents

Slide 13

Errors: the geometry breaks, the information does not

• When the monkeys err, context loses its abstract format but can still be decoded
• A: context CCGP drops on error trials in all three areas. B: decoding of context, with all conditions in training, does not drop significantly

Return to contents

Slide 14

Recap: measures of one population

Why it matters: both read the population's geometry, how its conditions are arranged. Shattering says how many new rules a downstream neuron could learn; CCGP says whether what it learned carries over to new situations, and it drops when the monkeys err.

Each score is one number about one population. When does such a number mean anything?

Return to contents

Slide 15

Baselines and tests

Return to contents

Slide 16

When does a score mean anything?

From the last lecture: correlation-distance RDMs of 51 object means, monkey IT and ResNet-18's late layer

• IT vs ResNet-18: RSA = 0.71. Is that high? A quantitative claim needs three things: is it statistically significant, is it reliable across samples of images, how close is it to its ceiling?

Return to contents

Slide 17

1. Is the score above chance?

What a meaningless pairing scores: shuffles and random baselines

Return to contents

Slide 18

Correlation by chance: the null distribution

• Unrelated data correlate a little by luck
• Null hypothesis: the two lists are unrelated. Two people rate 100 movies without watching them
• Their ratings still correlate by chance, up to about ±0.2; with 10 movies, ±0.6
• Null distribution: the spread of the score over many unrelated pairs
• A score is above chance if it beats the null distribution's 95th percentile: only 5% of unrelated pairs score higher

Return to contents

Slide 19

Above chance? Shuffle the images

Computed: THINGS-fMRI, one person's IT (3,720 voxels), 100 images; six networks; 1,000 shuffles

• Only the part of a score above its shuffled baseline comes from matching the images
• Permutation (shuffle) test: shuffle one system's image order, recompute the score, repeat 1,000 times. This builds the null distribution from the data themselves
• A score passes if it lies above the 95th percentile of the shuffled scores. Monkey IT vs ResNet-18 passes: 0.71 against 0.055. So does every trained network in the figure
• Dashed and dotted lines: another person's IT and the noise ceiling (later today)

Return to contents

Slide 20

Readouts: shuffle the labels

• Bao et al. (2020) IT, 480 neurons. Animate vs inanimate, fit on 102 images, tested on 306
• Shuffled labels: 1.00 on training, chance on test (0.56). So we need separate test images
• Curse of dimensionality: more neurons (480) than training images (102), so any labels fit. Deep networks too (Zhang et al. 2017)

Return to contents

Slide 21

Random directions in many dimensions

Computed: random Gaussian vectors, 4,000 pairs per dimension

• Within one system: each stimulus is a vector of $D$ unit responses. In 1,000-D, two random stimulus vectors are nearly perpendicular: mean $|\cos\phi|$ 0.026 (2-D: 0.64)
• More units than stimuli: all nearly perpendicular, so 102 images in 480 neurons fit any labels

Return to contents

Slide 22

Few stimuli: unrelated systems can score high

• Linear CKA compares two systems' Gram matrices (dot products between stimuli). Two unrelated random systems, 4,096 units: near 1 with 10 stimuli
• Few stimuli, many units: each Gram matrix is mostly its diagonal, alike in any two systems

Return to contents

Slide 23

2. Would it hold on new images?

Two kinds of new images: images the fit never saw (cross-validation), and another sample of images (bootstrap)

Return to contents

Slide 24

Hold out every image once: cross-validation

• 5-fold cross-validation (left): 5 folds; fit on four, test on the fifth, rotate. Every image is tested once, by a fit that never saw it
• Right: one IT site predicted from ResNet-18 (fit next lecture). Score R²: here the squared correlation of prediction and response ($r = 0.77$, $R^2 = 0.60$)
• One split can be lucky: one fold alone gives R² 0.53 to 0.64; all five together, 0.60

Return to contents

Slide 25

Leakage: keep the test images out of every step

• Leakage: the test images influence some step of the fit. The test score is then too optimistic
• Every step is fit on the training images only: the z-scoring mean and standard deviation, the PCA directions, any setting (such as a penalty strength), and the weights
• Small leak: z-scoring with the mean of all images. Large leak: choosing units or settings by their test score; then even pure noise can look predictive
• Split first, then fit every step on the training images

Note: in our assignments the IT recordings come already cleaned, with missing values filled and each unit centred and scaled, using all 1,224 images before any split. That is a small leak, accepted to keep the assignments simple. To publish, do these steps on the training images only.

Return to contents

Slide 26

Would the ranking hold on a new sample of images?

Computed: the same THINGS-fMRI data, 1,000 resamples

• Cross-validation tests a fit on new images. But any score, even RSA with no fit, depends on which images the study used
• Bootstrap: draw 100 of our 100 images with replacement (some twice, some never), and rescore every model; repeat 1,000 times
• Criterion: compare two networks resample by resample. One clearly wins if it scores higher in at least 95% of resamples. ResNet-50 vs CLIP: 77%; vs the untrained network: 99%
• Verdict: all trained networks beat the untrained one; none clearly beats another trained one

Return to contents

Slide 27

3. How high could the score be?

Noise in the recordings sets a ceiling

Return to contents

Slide 28

Brains are noisy, networks are not

• 30 repeats of one image: an IT electrode responds differently each time; a network unit does not
• Pan's mean (dashed) is below stalagmite's, yet pan comes out above it on 9 of 30 presentations
• Noise caps every score. The split-half reliability, how well the means of two halves of the repeats agree across images, says how much of the recording repeats

Return to contents

Slide 29

How high could a score be? The noise ceiling

Computed: IT electrodes of two monkeys, 100 THINGS images, 20 random splits per number of repeats

• A model's score: CKA between the model and the IT recording
• Its ceiling: the same CKA between two halves of the recording, how well the brain agrees with itself. More repeats per half, closer to 1
• ResNet-18 vs IT, all repeats: 0.19 (monkey N), 0.36 (F). The halves agree at 0.95 and 0.92: the model is far from the ceiling
• Read a score against its ceiling: 0.3 is near the top if the ceiling is 0.35

Return to contents

Slide 30

The same area in another brain

Design only: 100 THINGS images × 30 repeats in V1, V4 and IT of two monkeys (Papale et al. 2025)

• Score all pairs of recordings. Reference for "brain-like": the same area in another animal
• Report a score between its references: the shuffled floor, the noise ceiling, another brain

Return to contents

Slide 31

Change the measure, and the winner changes

Computed: monkey IT (Bao et al.) vs seven networks, 51 object means, units z-scored

• Seven networks ranked by similarity to monkey IT on the same 51 objects. Only the measure changes
• RSA puts CLIP first; CKA puts ResNet-18 first
• The top networks nearly tie, so changing the measure reorders them

Return to contents

Slide 32

Recap

• Before trusting a score: its shuffled floor, a test on held-out images, its noise ceiling and the same area in another brain; then resample the images and try more than one measure
• Next lecture: one model neuron, and how its weights are learned

Return to contents

Slide 33

Further reading for the research trail

• Bernardi et al. Cell 2020: flexible and abstract at once in hippocampus and prefrontal cortex, and the geometry breaks on error trials
• Schütt et al. eLife 2023: resampling images and participants to test a model ranking
• Safaie et al. Nature 2023: the same population dynamics in motor cortex across animals
• Papale et al. Neuron 2025: V1, V4 and IT in two monkeys, 30 repeats per image
• Zhang et al. ICLR 2017: deep networks fit random labels perfectly

Trail entry 2, due Fri 10/9: pick any paper cited in a lecture so far, read it, and post four to six sentences on Ed under Research trail: what it did, what surprised you or the question it left you with, and what you would test next. Reply once to a classmate. Best three of four entries count.

Return to contents

Slide 34

Minute paper

Submit on Canvas → Minute papers → Minute Paper 8 (access code read out in class).

Write three brief points in your own words:

1. A network's RSA with monkey IT is 0.20. Name three reference scores you would compute
   before calling it high or low, and say what each one tells you.
2. Something you do not yet understand, or a question still open
3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents