Representations, spaces & metrics

Reader-friendly version of the 45 lecture slides: the lecture content in normal flow, with figure descriptions and data tables where the figure carries data. Open the presentation.

Notation: whole vectors use bold lowercase letters; scalar components use regular italic letters; matrices use bold uppercase. A, B and C label objects in the worked examples.

Slide 1

Brown University crest
CPSY 1291 Computational Methods for Mind, Brain & Behavior
Lecture 1 · Theme 1: Representational spaces

Representations, spaces & metrics

Tuesday, September 15 · Fall 2026

Return to contents

Slide 2

Before we start

Return to contents

Slide 3

What does a brain or network represent?

Three tigers. Which two are most alike? Two systems, two answers. What does each look at?

Three tiger photographs A, B and C from the assignment image set. Below them, two rows of checks and crosses: one system pairs A with B; another pairs B with C. A and B are near mirror images of one another; B and C are both walking with the head lowered

Three tiger photographs A, B and C from the assignment image set. Below them, two rows of checks and crosses: one system pairs A with B; another pairs B with C. A and B are near mirror images of one another; B and C are both walking with the head lowered

Open full-size figure
Natural photographs from Peterson, Abbott & Griffiths Cognitive Science 2018 · ImageNet photographs; the image set you will use in Assignment 1

Return to contents

Slide 4

A representation stands for something

“This is not a pipe.”

  • A depiction is not the object
  • It stands for it: a representation
Magritte’s painting depicts a smoking pipe above the French inscription Ceci n’est pas une pipe, meaning This is not a pipe

René Magritte’s 1929 painting The Treachery of Images, at LACMA. A brown smoking pipe on a beige ground above a cursive French inscription, “Ceci n’est pas une pipe” (“This is not a pipe”). The painting separates an object from an image that depicts it. Photograph © Museum Associates/LACMA; artwork © C. Herscovici/Artists Rights Society (ARS), New York.

Open full-size figure

Return to contents

Slide 5

The representation in an image

Return to contents

Slide 6

Color images: three values per pixel

Vectorization: the 28 × 28 array is read row by row into a single list of 784 numbers, a vector. Distances and averages are defined on vectors.

Return to contents

Slide 7

Grayscale: one value per pixel

Vectorization: the 28 × 28 array is read row by row into a single list of 784 numbers, a vector. Distances and averages are defined on vectors.

Return to contents

Slide 8

One image, 784 measured values

Vectorization: the 28 × 28 array is read row by row into a single list of 784 numbers, a vector. Distances and averages are defined on vectors.

Return to contents

Slide 9

The representation in a brain

Return to contents

Slide 10

An image in human visual cortex

  • fMRI: blood-oxygenation signal per voxel
    • ~2 mm cube, thousands of neurons
  • Localizer scan: finds the regions
  • Each voxel's response to the dog: one entry of the vector
Top: cortical surfaces with early visual cortex in red and the lateral occipital complex in green, found by a localizer scan. Bottom: a dog photograph beside the measured responses of all 210 early visual and 152 LOC voxels of participant CSI1 to that photograph. Surface colors identify regions, not responses.

Top: cortical surfaces from Gao et al. (2025), Figure 6b, with early visual cortex in red and the lateral occipital complex (LOC) in green, defined by a localizer scan. The colors mark regions, not responses to the dog. Bottom: the ImageNet dog photograph shown in the experiment, beside the responses of all 210 early visual and 152 LOC voxels of participant CSI1, left hemisphere. Each bar is one voxel in stored order, and each value is the average of the third and fourth scans after onset. Each region gives one ordered response vector.

Open full-size figure
Numeric alternative to this figure
Participant CSI1: all left-hemisphere voxel responses to the dog photograph, in stored order (processed BOLD).
RegionVoxel index (1-based)Response
LHEarlyVis1-0.005631082314052525
LHEarlyVis2-0.02046481788947087
LHEarlyVis30.00026718470368442123
LHEarlyVis40.0013215602131918338
LHEarlyVis5-0.0072630922946105465
LHEarlyVis6-0.014286982174250073
LHEarlyVis70.002176354461083021
LHEarlyVis80.012855817341410992
LHEarlyVis9-0.0007554271373356361
LHEarlyVis100.018316789735022213
LHEarlyVis110.0034127970419153626
LHEarlyVis120.005901079563377872
LHEarlyVis13-0.010937670870366793
LHEarlyVis14-0.004528280762144942
LHEarlyVis150.005029421146779989
LHEarlyVis160.00967484820921544
LHEarlyVis170.002142839104041341
LHEarlyVis18-0.007886396099769327
LHEarlyVis190.01900453178698771
LHEarlyVis20-0.003680536712557714
LHEarlyVis21-0.00916957642172731
LHEarlyVis220.007564883006384638
LHEarlyVis230.006983017438219254
LHEarlyVis24-0.0008172506055407294
LHEarlyVis250.006040180356118708
LHEarlyVis26-0.006417995939855423
LHEarlyVis27-0.010435964805621603
LHEarlyVis28-0.013625327852469343
LHEarlyVis290.008688421609435152
LHEarlyVis300.0007499833923928882
LHEarlyVis31-0.0006853159726076781
LHEarlyVis320.019077786458874355
LHEarlyVis33-0.001846042184916868
LHEarlyVis340.04096570253592632
LHEarlyVis35-0.002998622638051775
LHEarlyVis36-0.013948994549065818
LHEarlyVis370.0025731990191723503
LHEarlyVis38-0.002262638414641833
LHEarlyVis390.006333070985337638
LHEarlyVis400.004028493330840558
LHEarlyVis41-0.01005927594754378
LHEarlyVis420.0042734527243799844
LHEarlyVis43-0.004018207704979338
LHEarlyVis440.008010935396422164
LHEarlyVis45-0.006784241192991989
LHEarlyVis46-0.004388090316792921
LHEarlyVis47-0.006858253169539509
LHEarlyVis48-0.012573277917313084
LHEarlyVis49-0.010443105497467974
LHEarlyVis50-0.007506113793442777
LHEarlyVis51-0.0028027661609228427
LHEarlyVis52-0.001077591786413715
LHEarlyVis53-0.014356370433647254
LHEarlyVis540.0005007087031235194
LHEarlyVis550.0040357844247968355
LHEarlyVis560.006302036208052146
LHEarlyVis570.0034293954075332407
LHEarlyVis580.0005189082140877573
LHEarlyVis590.0004683321781079402
LHEarlyVis60-0.019026480495960174
LHEarlyVis61-0.034665091038648045
LHEarlyVis620.0009334654642518441
LHEarlyVis63-0.008532874083695893
LHEarlyVis64-0.0013149363931009052
LHEarlyVis650.009173860808534995
LHEarlyVis660.0010112695389614436
LHEarlyVis67-0.010782111223610065
LHEarlyVis680.02339664149858725
LHEarlyVis690.013168595700567578
LHEarlyVis700.006702316505589117
LHEarlyVis71-0.003161154106892263
LHEarlyVis72-0.006014181974042328
LHEarlyVis730.0076578883581748195
LHEarlyVis740.0007442140276953983
LHEarlyVis750.008550487274480519
LHEarlyVis760.010499439689023355
LHEarlyVis77-0.0065126334785002
LHEarlyVis780.006659539887113424
LHEarlyVis790.024682942409885625
LHEarlyVis800.01341377553525846
LHEarlyVis810.004428661241995367
LHEarlyVis820.030302192892524945
LHEarlyVis830.023867674684985615
LHEarlyVis840.00048631142082219904
LHEarlyVis850.002056901712741485
LHEarlyVis860.004633134962552011
LHEarlyVis870.03230426372745228
LHEarlyVis88-0.008526930723591473
LHEarlyVis89-0.0002999057365226359
LHEarlyVis900.003747824376114307
LHEarlyVis91-0.002535085290730822
LHEarlyVis920.0027485899797380048
LHEarlyVis93-0.007167356104696165
LHEarlyVis94-0.0036148946700014437
LHEarlyVis95-0.009343848161819952
LHEarlyVis960.005141775829751127
LHEarlyVis970.0011671945602346788
LHEarlyVis98-0.002609883231827999
LHEarlyVis990.009130991971117589
LHEarlyVis1000.0046709016334143
LHEarlyVis1010.005986803664697348
LHEarlyVis1020.0043042870100837615
LHEarlyVis103-0.008098282609215749
LHEarlyVis104-0.005669559599971922
LHEarlyVis1050.00025368581278675155
LHEarlyVis1060.002622419037163583
LHEarlyVis1070.004325011094194638
LHEarlyVis1080.002595447265517789
LHEarlyVis1090.00774139779933595
LHEarlyVis1100.010724766907930016
LHEarlyVis1110.010276995282770216
LHEarlyVis1120.0063603604360014535
LHEarlyVis1130.01607757157794744
LHEarlyVis114-0.0013460825832159367
LHEarlyVis1150.0005020840315778823
LHEarlyVis1160.016708332817426244
LHEarlyVis1170.016964964877829995
LHEarlyVis1180.02406331505419822
LHEarlyVis1190.017734244222047627
LHEarlyVis1200.0030755638059260446
LHEarlyVis1210.014929258698051623
LHEarlyVis1220.008142583108806143
LHEarlyVis123-0.004287779905312508
LHEarlyVis124-0.0011461812973774415
LHEarlyVis1250.01196133150720466
LHEarlyVis1260.00036075676045555123
LHEarlyVis1270.004953942303107795
LHEarlyVis1280.0072830152692711005
LHEarlyVis1290.01185293937297482
LHEarlyVis1300.012953567106707555
LHEarlyVis1310.012791191630542566
LHEarlyVis1320.0052745596536178645
LHEarlyVis1330.02528801638566122
LHEarlyVis1340.0070315970240415245
LHEarlyVis1350.0012207779380217568
LHEarlyVis1360.007264586355114223
LHEarlyVis137-0.0012698292008318578
LHEarlyVis1380.0008584788759713532
LHEarlyVis1390.008619367802299188
LHEarlyVis1400.01683449346912593
LHEarlyVis1410.006405525037498289
LHEarlyVis1420.01275554961767062
LHEarlyVis143-0.00440573062761308
LHEarlyVis1440.0014738230664795128
LHEarlyVis1450.00729078665890947
LHEarlyVis146-0.00615760975221005
LHEarlyVis1470.0001632398211960939
LHEarlyVis1480.012542125655747666
LHEarlyVis1490.009171277715260037
LHEarlyVis1500.015812856772962243
LHEarlyVis1510.004116925989060969
LHEarlyVis1520.00553236010856187
LHEarlyVis1530.01105072949143213
LHEarlyVis1540.0016477100362305482
LHEarlyVis1550.012819464518688829
LHEarlyVis1560.007441718892039739
LHEarlyVis157-0.001472959418776391
LHEarlyVis1580.012148066917061214
LHEarlyVis1590.008778471370285767
LHEarlyVis1600.006233766635771309
LHEarlyVis161-0.003598524399124309
LHEarlyVis1620.00585653305525821
LHEarlyVis1630.0027924479343986435
LHEarlyVis1640.007623830453826506
LHEarlyVis1650.010502346842430642
LHEarlyVis1660.00940769574288116
LHEarlyVis1670.010025696386602646
LHEarlyVis1680.015621625660481829
LHEarlyVis1690.00576055271874518
LHEarlyVis170-0.002992975313500256
LHEarlyVis1710.004222706069552948
LHEarlyVis1720.01461096915781565
LHEarlyVis1730.027748958492125705
LHEarlyVis1740.01305910153782756
LHEarlyVis1750.016159761804854338
LHEarlyVis1760.008589080560494016
LHEarlyVis1770.005933517455620102
LHEarlyVis1780.006939206135248737
LHEarlyVis1790.005621900573716873
LHEarlyVis180-0.0024817234637845896
LHEarlyVis1810.017791615931256718
LHEarlyVis1820.00897423050722566
LHEarlyVis1830.01209132415825098
LHEarlyVis1840.0008222418238019531
LHEarlyVis185-0.0007871215954441844
LHEarlyVis1860.001340991700890245
LHEarlyVis1870.008756288132994169
LHEarlyVis1880.01186732835948223
LHEarlyVis1890.0067883244999451045
LHEarlyVis1900.003563938100731505
LHEarlyVis1910.011370028967857621
LHEarlyVis1920.006607829859153184
LHEarlyVis1930.0071097147951641555
LHEarlyVis1940.009917663122867425
LHEarlyVis1950.012321813225329136
LHEarlyVis1960.006287017475542736
LHEarlyVis1970.006798174899565619
LHEarlyVis1980.00792741142753157
LHEarlyVis1990.020945025266809667
LHEarlyVis2000.002167792208561634
LHEarlyVis2010.008210520974682113
LHEarlyVis2020.018994524250788028
LHEarlyVis2030.008588619203525969
LHEarlyVis2040.006434996897425769
LHEarlyVis2050.01242227027640869
LHEarlyVis2060.00817188629784936
LHEarlyVis207-0.0020694027956862484
LHEarlyVis208-0.001835446249872638
LHEarlyVis209-0.00641324390794939
LHEarlyVis2100.006913606034388514
LHLOC1-0.012388295701353101
LHLOC20.010661290629322018
LHLOC30.004227652828946073
LHLOC4-0.001693489834629156
LHLOC5-0.00786773936572782
LHLOC60.011790741257794262
LHLOC7-0.0000030486403119851443
LHLOC80.003937097626142938
LHLOC9-0.0019027390675707223
LHLOC100.0015380553990969074
LHLOC110.03372321811433824
LHLOC120.004929674784202793
LHLOC130.0046254888249637385
LHLOC140.00874991825055204
LHLOC15-0.00243288645530171
LHLOC16-0.004924201207757459
LHLOC170.007456964537142343
LHLOC180.005736288912875373
LHLOC190.0004046690592518277
LHLOC20-0.0046243111408615645
LHLOC210.006825754187071346
LHLOC220.0077207709657446215
LHLOC230.008844240690559075
LHLOC240.006138978864577404
LHLOC250.013710569203134558
LHLOC260.012751877582669753
LHLOC270.010394292454052672
LHLOC28-0.000010885800925610097
LHLOC290.0006428218731120115
LHLOC300.010829757288835938
LHLOC310.02220321310801266
LHLOC320.004544683485686354
LHLOC330.047215788292589565
LHLOC340.030052529381133723
LHLOC35-0.0003903124774146103
LHLOC36-0.004372962010418639
LHLOC37-0.006113776264562207
LHLOC38-0.0031145193373918047
LHLOC390.0015749921419302107
LHLOC400.029897359264332207
LHLOC410.007676076264343917
LHLOC42-0.002099162565982684
LHLOC430.002022999077082879
LHLOC440.007155690669035649
LHLOC450.023848968530586788
LHLOC460.017000958191089592
LHLOC470.011108131116827636
LHLOC480.004690723970917843
LHLOC490.008999137498024548
LHLOC500.0009346352072650871
LHLOC510.005131151823294625
LHLOC520.011492044212828902
LHLOC53-0.0025930793781772103
LHLOC540.015224407234208122
LHLOC550.004907124173406824
LHLOC560.007446494358064769
LHLOC57-0.009422177161197713
LHLOC580.006932666548603259
LHLOC590.02640018230705267
LHLOC600.020343314951088737
LHLOC610.008570833059347898
LHLOC62-0.005502264486356734
LHLOC630.006529175936157134
LHLOC640.0053078856544283
LHLOC650.013289408297409783
LHLOC660.007544249465121516
LHLOC670.002551896534432001
LHLOC68-0.003689386623800919
LHLOC690.006366205286871145
LHLOC700.03375395112521438
LHLOC710.01057295821665236
LHLOC720.0016207491382084626
LHLOC73-0.0008144428021264875
LHLOC740.00901784296928983
LHLOC750.012208217152606431
LHLOC760.015354217647067453
LHLOC770.0029393430391396074
LHLOC780.010485011213475837
LHLOC790.004452081583201901
LHLOC800.008709273823417865
LHLOC810.002344840278724152
LHLOC820.012071457494324334
LHLOC830.006563230835283097
LHLOC840.0018093121735980474
LHLOC850.0011444899544917457
LHLOC860.022950119979867685
LHLOC870.00033654344864311975
LHLOC880.01194960681369199
LHLOC890.001629068091218868
LHLOC900.009728317480210639
LHLOC910.015898362351150597
LHLOC920.0009478292644730007
LHLOC930.0073763081127763905
LHLOC940.0038659341264463414
LHLOC950.0019566864992922725
LHLOC960.006996823351562674
LHLOC970.005451584482105372
LHLOC98-0.00814744977132911
LHLOC99-0.002223820035718061
LHLOC1000.014232393385626149
LHLOC1010.01745037175540831
LHLOC102-0.00026425696974702057
LHLOC1030.025891205797634268
LHLOC1040.005876041446209417
LHLOC105-0.005495536151664165
LHLOC1060.003386891767405943
LHLOC1070.001654311695809507
LHLOC1080.011238956456255124
LHLOC1090.015875854907499758
LHLOC110-0.0023781046413575966
LHLOC1110.010435700589326255
LHLOC112-0.00016733887310885816
LHLOC113-0.0019279326942263936
LHLOC1140.018858265029703947
LHLOC1150.02572137922159229
LHLOC1160.018969382302698377
LHLOC1170.0067689867092788885
LHLOC1180.011677739388864407
LHLOC1190.001965234112930835
LHLOC1200.00013387145953864393
LHLOC1210.0018242440315302342
LHLOC1220.02791540749553585
LHLOC1230.0052287597977691384
LHLOC1240.015916395752873527
LHLOC1250.007972180324508714
LHLOC1260.00863035845370602
LHLOC127-0.013991365587773702
LHLOC1280.010270469042001516
LHLOC1290.007290289215369657
LHLOC1300.005089516505477872
LHLOC131-0.0016206740795654522
LHLOC1320.004772698573029304
LHLOC133-0.008718528594982239
LHLOC134-0.0008563969820974079
LHLOC1350.009461504779807593
LHLOC1360.0007316418478594243
LHLOC137-0.0026861700975371427
LHLOC1380.013834132429931357
LHLOC1390.010306009992901592
LHLOC1400.02402721643572683
LHLOC1410.011685655260489959
LHLOC1420.008485240373731485
LHLOC1430.0002860689110119291
LHLOC1440.003929729761755232
LHLOC145-0.008579545167773912
LHLOC146-0.004456010437859775
LHLOC147-0.00035681290778699236
LHLOC1480.008359332803091854
LHLOC1490.008110145700793867
LHLOC1500.008876587920013
LHLOC151-0.00010691416594510277
LHLOC1520.008935419791845558

Return to contents

Slide 11

Visual selectivity, heard as spikes

Hubel & Wiesel · cat V1 · Historical context: Wiesel, iBiology

Return to contents

Slide 12

One electrode at a time, or hundreds of sites at once

A single metal microelectrode, a thin needle, mounted in a holder that advances it into the brain

A single metal microelectrode, a thin needle, mounted in a holder that advances it into the brain

Open full-size figure

A single microelectrode: one site

The Neuropixels probe: a schematic of the shank tip with two staggered columns of square recording sites 20 micrometres apart on a 70-micrometre-wide shank; a scanning electron micrograph of the tip; and the whole device, a 1 centimetre shank rising into a base, flex cable and headstage

The Neuropixels probe: a schematic of the shank tip with two staggered columns of square recording sites 20 micrometres apart on a 70-micrometre-wide shank; a scanning electron micrograph of the tip; and the whole device, a 1 centimetre shank rising into a base, flex cable and headstage

Open full-size figure

Neuropixels: 960 sites on one shank

  • Bao et al. 2020, the monkey data of this course: a tungsten microelectrode advanced into IT, one neuron at a time; 482 neurons over many sessions
  • Neuropixels (Jun et al. 2017): 960 sites, 384 read at once, 20 µm apart
    • One neuron appears on several sites; that pattern is what spike sorting uses
    • One insertion: hundreds of neurons, several areas
  • Same kind of number either way: a firing rate per neuron
Microelectrode photograph: Alpha Omega Engineering · Jun et al. Nature 2017 · Fig. 1 · Steinmetz et al. Science 2021 (Neuropixels 2.0: 5,120 sites)

Return to contents

Slide 13

Recording neural populations

Montage of neural recording technologies: (a) a rodent head in cross-section showing scalp EEG screws, an ECoG grid on the cortical surface, and a microelectrode entering the tissue; (b) the signal chain and example traces from EEG, ECoG, local field potential, and extracellular action potentials; (c) a flexible ECoG grid beside a coin; (d) a Utah array of one hundred silicon needles; (e) a silicon probe shank with dense recording sites; (f) shanks carrying micro-LEDs; (g) a micro-ECoG surface array over blood vessels; (h) a probe with on-board CMOS electronics; (i) a fluidic probe with electrodes and channels; (j) a small three-dimensional array with flexible interconnect

Montage of neural recording technologies: (a) a rodent head in cross-section showing scalp EEG screws, an ECoG grid on the cortical surface, and a microelectrode entering the tissue; (b) the signal chain and example traces from EEG, ECoG, local field potential, and extracellular action potentials; (c) a flexible ECoG grid beside a coin; (d) a Utah array of one hundred silicon needles; (e) a silicon probe shank with dense recording sites; (f) shanks carrying micro-LEDs; (g) a micro-ECoG surface array over blood vessels; (h) a probe with on-board CMOS electronics; (i) a fluidic probe with electrodes and channels; (j) a small three-dimensional array with flexible interconnect

Open full-size figure
a  Scales of access: scalp EEG → ECoG on the cortical surface → microelectrodes in the tissue
b  Signal chain: EEG and ECoG average many neurons; only electrodes in the tissue resolve spikes
c  ECoG grid — a flexible surface array (coin for scale)
d  Utah array — 100 silicon needles, 400 µm apart; used in human BCI
e  Silicon probe — dozens of recording sites along one shank (Neuropixels lineage)
f  Optoelectrode — micro-LEDs on the shank for optogenetics
g  Micro-ECoG — surface array fine enough for single cells
h  CMOS probe — amplifiers built into the probe base
i  Fluidic probe — electrodes plus channels for drug delivery
j  3D active array — flexible interconnect

Return to contents

Slide 14

From image to response: present an image

  • Image shown for 250 ms (Bao et al. 2020); signal travels retina → inferotemporal cortex (IT)

Return to contents

Slide 15

From image to response: record and sort spikes

  • Spike: a brief electrical pulse from a neuron. An electrode measures it; spike sorting assigns each spike to one neuron

Return to contents

Slide 16

From image to response: estimate a firing rate

  • Count spikes in a window, divide by its length: 7 / 0.160 s = 43.75 spikes/s. One number per neuron, per image, per trial

Return to contents

Slide 17

One image, several trials: average the rates

  • The same image is shown several times; each trial gives a different count. The response we keep is the mean rate across trials
Six repeated presentations of the tiger: six spike counts in the 60 to 220 ms window (5, 8, 6, 9, 7, 7), six rates from 31.25 to 56.25 spikes per second, and their mean, 43.75 spikes per second

Six repeated presentations of the tiger: six spike counts in the 60 to 220 ms window (5, 8, 6, 9, 7, 7), six rates from 31.25 to 56.25 spikes per second, and their mean, 43.75 spikes per second

Open full-size figure
Illustrative counts · Bao et al. Nature 2020 averaged 4–8 presentations per image

Return to contents

Slide 18

An object in a neural population

Left: Bao Fig. 4c coronal MRI showing functional IT patches, and a cropped Extended Data 8a recording-site detail with a 5 mm scale bar. Right: the framed cat stimulus above a bar plot of firing rates in spikes per second for all 482 units, 479 available values and three unavailable entries marked below the axis

Left: a coronal MRI from Bao et al. (2020), Figure 4c, showing functionally localized IT patches in monkey M3, and a cropped detail from Extended Data Figure 8a marking three example recording sites in the Stubby2 patch, with a 5 mm scale bar. The two views are separate and do not show the locations of all 482 neurons. Right: the monochrome cat stimulus in a light frame, above a bar plot of firing rates for all 482 units. The vertical axis runs from 0 to 100; the largest value is about 80.75. There are 479 available values; small marks below the axis flag missing measurements at positions 429, 430 and 457, which are not zeros. Bao recorded every well-isolated neuron encountered at the targeted sites, with no selectivity screen. The experiment used 1,224 rendered views, each shown for 250 ms and repeated 4–8 times; activity was counted 60–220 ms after onset.

Open full-size figure
Numeric alternative to this figure
All 482 unit positions for the Bao cat stimulus: 479 values; positions 429, 430 and 457 are missing measurements.
Unit (1-based)Response (shared value)
10
220.7039
34.65839
434.1615
52.07039
60
76.21118
80
98.28157
1018.6335
1116.5631
1226.9151
1324.8447
146.21118
1516.5631
160
179.31677
1831.0559
194.65839
206.21118
210
2214.4928
2317.0807
2415.528
2512.4224
268.28157
2720.7039
2810.352
298.28157
3022.7743
317.76398
3221.7391
330
3410.8696
354.65839
364.96894
3724.8447
382.07039
391.24224
406.21118
413.10559
423.10559
4318.6335
4455.9006
450
4618.6335
4747.619
482.07039
496.21118
5024.8447
5110.352
5221.7391
530
5424.8447
556.21118
5618.6335
573.10559
5812.4224
596.21118
600
6131.0559
620
630
6462.1118
653.10559
6659.0062
6739.3375
6843.4783
6924.8447
7015.528
714.14079
7212.4224
730
748.28157
750
760
774.14079
7826.9151
7914.4928
8021.7391
8127.9503
826.21118
8331.0559
8418.6335
853.10559
8622.7743
876.21118
880
899.31677
9024.8447
910
9210.352
936.21118
9449.6894
9526.9151
968.28157
978.28157
980
998.28157
1007.45342
10112.4224
1024.65839
10312.4224
1046.21118
10523.6025
1066.21118
10726.087
10826.3975
1099.31677
11010.8696
11126.3975
1126.21118
11310.352
1140
11524.8447
1167.45342
11716.5631
1187.76398
1193.10559
1201.5528
1210
1228.28157
12321.7391
12418.6335
1252.07039
12610.352
12722.7743
12814.4928
12937.2671
13010.352
1310
13212.4224
13335.1967
1344.14079
1353.10559
1363.10559
1370
13835.1967
1394.14079
1406.21118
1418.28157
14218.6335
14324.8447
14433.1263
1452.07039
1460
1478.28157
14824.8447
1490
1500
1516.21118
1520
1530
1540
1556.21118
1562.07039
1572.07039
1580
1596.21118
1603.10559
16118.6335
16238.8199
1634.14079
16414.4928
1657.76398
16649.6894
16718.6335
1680
1690
17019.8758
17153.8302
17212.4224
1732.07039
17412.4224
17513.6646
17614.9068
17739.3375
17820.7039
17920.7039
18010.352
1812.07039
18218.6335
1839.31677
1842.07039
1858.28157
1860
18721.7391
18833.1263
1894.14079
1908.28157
1910
1920
1931.5528
1944.65839
1954.65839
19620.7039
19720.1863
19817.0807
19918.6335
2006.21118
20137.2671
20212.4224
2037.76398
2046.21118
2050
2064.14079
2071.24224
2082.48447
2091.24224
2100
2110
21212.4224
2139.31677
21415.528
21512.4224
2163.10559
2173.10559
2181.5528
2198.28157
2204.14079
2216.21118
22217.0807
2231.5528
22428.9855
2256.21118
2262.07039
22714.4928
2280
22912.4224
23010.8696
23131.0559
2323.10559
23326.9151
2343.10559
2351.5528
23615.528
23710.8696
23826.3975
2394.65839
24026.3975
24137.2671
24210.8696
24318.6335
24431.0559
2456.21118
24637.2671
2473.10559
24823.2919
24937.2671
25074.5342
25132.6087
2523.10559
25362.1118
25421.7391
25518.6335
25620.1863
2577.76398
25813.9752
25931.0559
26068.323
26180.7453
2628.28157
26355.9006
26426.9151
26516.5631
26628.9855
2673.10559
2682.54094
2693.10559
27021.7391
2713.10559
27231.0559
27328.9855
2743.10559
2756.21118
27628.9855
27718.6335
27824.8447
27912.4224
28024.8447
28118.6335
28218.6335
28315.528
2849.31677
28510.8696
28617.0807
28713.9752
28834.1615
2898.69565
29011.1801
2917.76398
29240.3727
2936.21118
2946.21118
29512.4224
29612.4224
29724.8447
29814.4928
2996.21118
30012.4224
30123.2919
30228.9855
30321.7391
3048.28157
3059.31677
30631.0559
30745.5487
30862.1118
30966.2526
31026.3975
31154.3478
31233.1263
3134.14079
31477.6398
31552.795
31624.8447
31745.5487
31845.5487
31915.528
32010.8696
32123.2919
3224.65839
3233.10559
3241.5528
3250
3260
32740.3727
3281.5528
3290
3300
33126.9151
33220.7039
3336.21118
33410.352
3354.14079
33624.8447
3372.07039
33818.6335
33922.7743
34012.4224
34112.4224
3423.10559
34323.2919
34424.8447
3459.31677
3466.21118
3472.07039
34814.4928
34922.7743
35010.352
35110.352
3528.28157
35332.6087
3540
3556.21118
35619.8758
35731.0559
35813.9752
35924.8447
36015.528
36113.9752
3629.31677
3633.10559
36413.9752
36540.3727
36616.5631
3670
3686.21118
3697.76398
37051.2422
37131.0559
37217.0807
37333.1263
37418.6335
37549.6894
3760
3770
3780
3794.14079
3800
3813.10559
38212.4224
38310.352
3843.10559
3858.28157
3861.5528
3872.07039
3880
38918.6335
39020.1863
3914.65839
3924.65839
39315.528
3943.10559
3958.28157
3960
3976.21118
3980
3992.07039
4008.28157
4014.14079
4027.76398
40315.528
4040
4054.65839
4060
4070
4082.07039
4090
4100
4119.31677
41212.4224
4130
4140
4150
4164.14079
41718.6335
4184.14079
41943.4783
4204.14079
4216.21118
4224.14079
4236.21118
4243.10559
4256.21118
4264.14079
4272.07039
4286.21118
429Unavailable (NaN in source)
430Unavailable (NaN in source)
43115.528
4326.21118
43312.4224
4348.28157
4356.21118
43620.7039
43724.8447
4386.21118
4392.07039
4400
4416.21118
4421.5528
4430
4440
4450
4460
4476.21118
44821.7391
4490
4502.07039
4510
4526.21118
4536.21118
4543.10559
4552.07039
45610.352
457Unavailable (NaN in source)
4583.10559
4590
4608.28157
46126.9151
4620
4632.07039
4644.14079
4658.28157
4660
4672.07039
4680
4690
4700
4712.07039
4720
4730
4744.14079
4750
4760
4771.24224
47824.8447
4798.28157
4806.21118
4814.14079
4824.96894
Bao et al. Nature 2020 · IT patches: Fig. 4c · recording sites: Extended Data Fig. 8a · responses: the Assignment 1 data

Return to contents

Slide 19

The representation in an artificial neural network

Return to contents

Slide 20

Visual cortex and convolutional networks

Macaque visual areas above a hierarchical CNN with stacks of spatial feature maps; dashed green arrows mark proposed model–brain correspondences

Figure 1b–c of Yamins and DiCarlo (2016): ventral-stream areas above a candidate hierarchical convolutional model. A macaque brain inset locates V1, V2, V4 and posterior, central and anterior IT; the upper pathway also includes retinal ganglion cells (RGC) and the lateral geniculate nucleus (LGN). Arrows connect successive populations, including recurrent and feedback connections. Below, stacked sheets represent the feature maps at successive network layers; small red boxes mark local receptive fields, and a lower-right inset lists filtering, thresholding, pooling and normalization. Green dashed arrows mark proposed correspondences between model stages and brain areas, to be tested with neural data. The network is schematic, and the 100-ms presentation belongs to the source figure; Bao's images were shown for 250 ms.

Open full-size figure

Return to contents

Slide 21

From AlexNet to ResNet

ImageNet top-5 accuracy by year: AlexNet 83.6 percent with 8 layers in 2012, ZFNet 88.3 with 8 layers in 2013, GoogLeNet 93.3 with 22 layers and VGG-16 92.7 with 16 layers in 2014, ResNet-152 96.4 with 152 layers in 2015

ImageNet top-5 accuracy by year: AlexNet 83.6 percent with 8 layers in 2012, ZFNet 88.3 with 8 layers in 2013, GoogLeNet 93.3 with 22 layers and VGG-16 92.7 with 16 layers in 2014, ResNet-152 96.4 with 152 layers in 2015

Open full-size figure
A mosaic of hundreds of small ImageNet photographs of animals, objects, vehicles and scenes

A mosaic of hundreds of small ImageNet photographs of animals, objects, vehicles and scenes

Open full-size figure
  • ImageNet: 14 M photographs, 22 k nouns
  • Benchmark: 1.28 M training images, 1,000 categories
    • Top-5: label among the five best guesses
  • Deeper every year. These models feed Assignment 1
  • Browse: the 1,000 categories · LENS, what a ResNet-50 learned, concept by concept.

Return to contents

Slide 22

Early, middle and late representations

The tiger photograph and three rows of six activation maps from ResNet-18: early-stage 56 by 56 maps that trace edges and stripes, middle-stage 14 by 14 maps that pick out parts and textures, and late-stage 7 by 7 maps that light up over the tiger's body

The tiger photograph and three rows of six activation maps from ResNet-18: early-stage 56 by 56 maps that trace edges and stripes, middle-stage 14 by 14 maps that pick out parts and textures, and late-stage 7 by 7 maps that light up over the tiger's body

Open full-size figure
He et al. CVPR 2016 · LENS (Serre lab; what a ResNet-50 learned, class by class — tiger)

Return to contents

Slide 23

An object in ResNet-18

Return to contents

Slide 24

Representations as measurements

A representation: a pattern that carries information about a stimulus, a property or a state. Each measurement so far gives one such pattern as a list of numbers.

Return to contents

Slide 25

Notation

Notation — table
Symbol Meaning
𝐱\mathbf{x} (bold lowercase) a vector: the whole list of numbers for one object
xjx_j (italic, subscript) its jjth component, a single number
𝐱(i)\mathbf{x}^{(i)}, 𝐱(i,r)\mathbf{x}^{(i,r)} the vector for image ii; the same image on repetition rr
𝐗\mathbf{X} (bold uppercase) a matrix: many vectors stacked, one image per row
xijx_{ij} its entry in row ii, column jj: feature jj of image ii
𝐱⊤\mathbf{x}^{\top} the transpose: the same numbers laid out as a row instead of a column
∥𝐱∥\lVert\mathbf{x}\rVert the length of the vector
DD, NN the number of measurements per object; the number of objects

Return to contents

Slide 28

Two measurements, two coordinates

x1x_1: mean intensity · x2x_2: RMS contrast (standard deviation of the intensities)

One vector, two components:

𝐱=(x1, x2)\mathbf{x}=(x_1,\,x_2)

Every image is a point in the same space. Comparing representations therefore reduces to geometry: distances and angles between points.

The same plot with all six images shown as thumbnails at their points: tiger, gorilla, eagle, frog, penguin and elephant

The same plot with all six images shown as thumbnails at their points: tiger, gorilla, eagle, frog, penguin and elephant

Open full-size figure

Return to contents

Slide 29

A vector is an arrow from the origin

Same two numbers: a point, or an arrow from the origin to it.

𝐱(1)≈(72.740.6)(the tiger)\mathbf{x}^{(1)}\approx\begin{pmatrix}72.7\\40.6\end{pmatrix}\quad\text{(the tiger)}

  • Vectors are columns; inline (x1, x2)(x_1,\,x_2) is shorthand
  • Superscript (1)(1): image number 1
  • Arrows: what lets us add and subtract vectors, measure similarity, dissimilarity and distance, and so compare representations
The same square plot of RMS contrast against mean intensity with a single arrow drawn from the origin to the tiger's point at 72.7 and 40.6, labelled x superscript 1; the other five images are grey dots

The same square plot of RMS contrast against mean intensity with a single arrow drawn from the origin to the tiger's point at 72.7 and 40.6, labelled x superscript 1; the other five images are grey dots

Open full-size figure

Return to contents

Slide 30

Extending beyond two dimensions

𝐱(i)=(x1(i)x2(i)⋮xD(i))∈ℝD\mathbf{x}^{(i)}=\begin{pmatrix}x_1^{(i)}\\x_2^{(i)}\\\vdots\\x_D^{(i)}\end{pmatrix}\in\mathbb{R}^{D}

  • 𝐱(i)\mathbf{x}^{(i)}: representation of image ii, one number per measurement
  • DD: the dimension, the number of measurements
    • ℝD\mathbb{R}^{D}: all such lists
  • DD measurements define a DD-dimensional space; same object, different measurement → different space
Extending beyond two dimensions — table
Measurement DD
Two image features 2
Resampled grayscale image 784
Recorded IT population 482
Human early visual cortex (voxels) 210
Network layer (sampled units) 4,096

Return to contents

Slide 31

How reliable is one response?

Return to contents

Slide 32

Brain responses are noisy

  • Same image three times → three different vectors
  • Correlation rr: agreement, 1 = identical, 0 = unrelated (defined next lecture)
  • r12=r_{12}= 0.17, r13=r_{13}= −0.01, r23=r_{23}= 0.03; median over 112 repeated images 0.14
  • Averaging keeps what repeats
The bullfrog photograph shown three times to participant CSI1

The bullfrog photograph shown three times to participant CSI1

Open full-size figure
Four rows of bar plots over 210 early-visual-cortex voxels: three single presentations of the same photograph to participant CSI1, each visibly different, and below them in red the mean of the three

Four rows of bar plots over 210 early-visual-cortex voxels: three single presentations of the same photograph to participant CSI1, each visibly different, and below them in red the mean of the three

Open full-size figure
Data: Chang et al. Scientific Data 2019 · BOLD5000 CSI1, image n01641577_1229 (bullfrog), early visual region, average of two post-stimulus time points

Return to contents

Slide 33

Electrophysiology is noisy too; averaging works

  • Monkey IT, 168 sites, same image 51 times
  • Two single trials: r=r= 0.17
  • Split-half correlation: mean of 25 trials vs mean of the other 25, r=r= 0.69
  • Every response vector in this course is such a mean
The stimulus: a rendered lioness on a natural background, one of the 3,200 images of Majaj et al. 2015

The stimulus: a rendered lioness on a natural background, one of the 3,200 images of Majaj et al. 2015

Open full-size figure
Four rows of bar plots over 168 IT sites: three single presentations of the same image, each different, and below them in red the mean over all 51 presentations, much smaller and smoother

Four rows of bar plots over 168 IT sites: three single presentations of the same image, each different, and below them in red the mean over all 51 presentations, much smaller and smoother

Open full-size figure
Data: Majaj, Hong, Solomon & DiCarlo J. Neurosci. 2015, public Brain-Score assembly; responses are normalised per site, 70–170 ms window

Return to contents

Slide 34

Average repeated responses to one image

Repetition rr of RR gives 𝐱(i,r)∈ℝD\mathbf{x}^{(i,r)}\in\mathbb{R}^{D}. Three neurons, three repetitions (illustrative spikes/s):

𝐱(i,1)=(251),𝐱(i,2)=(432),𝐱(i,3)=(343)⟹𝐱(i)=1R∑r=1R𝐱(i,r)=(342)\mathbf{x}^{(i,1)}=\begin{pmatrix}2\\5\\1\end{pmatrix},\qquad \mathbf{x}^{(i,2)}=\begin{pmatrix}4\\3\\2\end{pmatrix},\qquad \mathbf{x}^{(i,3)}=\begin{pmatrix}3\\4\\3\end{pmatrix} \qquad\Longrightarrow\qquad \mathbf{x}^{(i)}=\frac{1}{R}\sum_{r=1}^{R}\mathbf{x}^{(i,r)}=\begin{pmatrix}3\\4\\2\end{pmatrix}

Return to contents

Slide 35

Column vectors and row vectors

Column: DD rows, one column. Transpose ⊤^{\top}: the same numbers as a row.

𝐱(i)=(342)∈ℝ3×1→ ⊤ (𝐱(i))⊤=(342)∈ℝ1×3\mathbf{x}^{(i)}=\begin{pmatrix}3\\4\\2\end{pmatrix}\in\mathbb{R}^{3\times 1} \qquad\xrightarrow{\ \top\ }\qquad \big(\mathbf{x}^{(i)}\big)^{\top}=\begin{pmatrix}3&4&2\end{pmatrix}\in\mathbb{R}^{1\times 3}

Return to contents

Slide 36

Many images form a response matrix

Stack NN row vectors: one matrix for the whole image set.

𝐗=((𝐱(1))⊤(𝐱(2))⊤⋮(𝐱(N))⊤)∈ℝN×D\mathbf{X}=\begin{pmatrix}\big(\mathbf{x}^{(1)}\big)^{\top}\\\big(\mathbf{x}^{(2)}\big)^{\top}\\\vdots\\\big(\mathbf{x}^{(N)}\big)^{\top}\end{pmatrix}\in\mathbb{R}^{N\times D}

  • Row ii: image ii. Column jj: feature jj
  • xij=xj(i)x_{ij}=x_j^{(i)}: feature jj of image ii, one cell
  • Here: N=6N=6 images, 12 of D=4,096D=4{,}096 activations shown

𝐗=(x11x12⋯x1Dx21x22⋯x2D⋮⋮⋮xN1xN2⋯xND)\mathbf{X}=\begin{pmatrix}x_{11}&x_{12}&\cdots&x_{1D}\\x_{21}&x_{22}&\cdots&x_{2D}\\\vdots&\vdots&&\vdots\\x_{N1}&x_{N2}&\cdots&x_{ND}\end{pmatrix}

Heatmap of six named animal images by twelve late-layer ResNet-18 activations; each cell is one entry x i j

Rows are six animal photographs; columns are the first 12 stored network activations. Each colored cell is the activation of one entry for one image, and a row is that image's response vector. The six by 4,096 matrix is shown in part; color encodes activation, not similarity.

Open full-size figure
Numeric alternative to this figure
Six animal images × first 12 stored late ResNet-18 activations; values rounded to six significant digits.
Image123456789101112
Frog01.506321.916240.8085121.472990.1751540.1406490.40427102.334111.21670
Eagle0.4583882.9282203.3124601.02485002.552150.37562700
Elephant000002.361110.8935811.197510001.68677
Tiger1.799691.41010.7158381.17101001.333921.8153801.415075.863630
Penguin0.1934986.311891.435321.323580.08196311.58981.486350.9965420.49173601.839320
Gorilla03.949471.6208501.73141000.078079801.2064102.261

Return to contents

Slide 37

Comparing representations across systems

Pixels, voxels, neurons and network units cannot be compared entry by entry: the vectors differ in length and in units. Their pairwise dissimilarities can. The RDM records them.

Return to contents

Slide 38

An RDM compares every pair of images

Representational dissimilarity matrix (RDM): every pairwise comparison.

𝐃∈ℝN×N,dik=d(𝐱(i),𝐱(k))\mathbf{D}\in\mathbb{R}^{N\times N},\qquad d_{ik}=d\bigl(\mathbf{x}^{(i)},\mathbf{x}^{(k)}\bigr)

  • Rows and columns: images, here ordered by category. Each entry: one comparison
  • dd: the dissimilarity measure. It decides which differences count
  • Entries are not arbitrary: dii=0d_{ii}=0, dik=dkid_{ik}=d_{ki}, and for a distance, no shortcut beats the direct route (next lecture)
A 120 by 120 representational dissimilarity matrix of the Assignment 1 images from the trained ResNet-18 late layer, correlation distance, images ordered by animal category with category boundaries drawn; light blocks along the diagonal show that primates, carnivores and hoofed mammals are more alike within their category than across

A 120 by 120 representational dissimilarity matrix of the Assignment 1 images from the trained ResNet-18 late layer, correlation distance, images ordered by animal category with category boundaries drawn; light blocks along the diagonal show that primates, carnivores and hoofed mammals are more alike within their category than across

Open full-size figure

Return to contents

Slide 39

Want to see inside a network before we get there?

Two lectures on convolutional networks and one on training are coming. Previews:

Not required. Ten minutes well spent: the explainer and the playground.

Return to contents

Slide 40

Further reading for the research trail

Six starting points, none required. Your trail paper can come from any lecture; these are seeds, not a list to choose from. Peterson et al. 2018 and Bao et al. 2020 are the assignment papers: fine to read, but branch out.

Return to contents

Slide 41

How to read for the trail

Return to contents

Slide 42

Minute paper

Canvas submission: Canvas → Minute papers → Minute Paper 2 (access code read out in class)

Write three brief points in your own words:

  1. One sentence: when can two response vectors have cosine similarity 1 (cosine dissimilarity 0) and yet a large Euclidean distance?
  2. Something you do not yet understand, or a question still open
  3. Another idea you found interesting, and why it matters for brains, behavior or AI

Credit for a thoughtful attempt, not for being correct.

Return to contents

Slide 43

Reference: computing an RDM

import numpy as np from scipy.spatial.distance import pdist, squareform X = np.array([[1., 1.], # pattern A [4., 4.], # pattern B [1., -1.]]) # pattern C for metric in ("euclidean", "cosine"): D = squareform(pdist(X, metric=metric)) print(metric, np.round(D, 2))

pdist: the unique pairs. squareform: the symmetric matrix.

Return to contents

Slide 44

Appendix · Scaling image intensity

Scaling: 𝐱↦c 𝐱\mathbf{x}\mapsto c\,\mathbf{x}. Same 784 entries, only their size changes.

Return to contents

Slide 45

Appendix · Averaging and blending images

Blending: 𝐳=(1−λ) 𝐱(1)+λ 𝐱(2)\mathbf{z}=(1-\lambda)\,\mathbf{x}^{(1)}+\lambda\,\mathbf{x}^{(2)}, entry by entry. Done to labels too, this is mixup.

Return to contents