All 144 heads of GPT-2 small
sentence:
strongest:
All 144 heads, one cell per head
block 0–11 down · head 0–11 across · color:
induction weight:01 ★ = strongest
Previous-token weight
Mean weight on the token just before, over 20 ordinary sentences.
Induction weight
In a random sequence shown twice, mean weight on the token after the earlier copy of the current token.
First-token weight
Mean weight on the first token, over the same 20 sentences.
Knock this head out
Its output set to 0; the loss measured again on two tests.
Copying loss: (intact )
Ordinary-text loss: (intact )
Losses in nats per token; bars show the increase, both on one scale (0 to ).