key kscore q·k÷√2weight
query q =
output = (weighted average of the values)
Thicker key arrow: more weight. Gray key: a later word, hidden by the causal mask. "asking" picks whose query is used.
Drag the magenta arrow. Try:
  1. Make sat read mostly the.
  2. Keep the direction, double the length. What happens to the weights?
  3. Ask from cat with the mask on, then off.