In self-attention, each token produces a query, a key and a value. Comparing a token’s query with every key gives attention weights, which decide how much of each value is mixed into the token’s new representation. Many attention “heads” run in parallel and learn different relationships, such as grammar or references.
Attention is the core of the Transformer. Its cost grows with the square of the sequence length, which is why long context windows are expensive and why researchers develop more efficient attention variants.