Intermediate8 min read

What is Self-Attention?

Self-attention is attention applied within one sequence — every word looks at every other word in the same sentence to build its meaning.

Explain like I'm new to AI

Regular attention can look from one sequence to another (like matching a question to a document). Self-attention is the special, most important case: the sequence looks at itself.

Every word simultaneously asks, "Which other words in this same sentence help me understand my role?" — and updates itself with that context. Do this for all words at once, and the whole sentence becomes context-aware in a single step.

Click any word to see what it pays attention to. Darker = stronger attention.

Query word: it — notice "it" attends most strongly to "animal", resolving what "it" refers to.

Attention lets each word gather context from the most relevant other words.

Mental model

Picture everyone in a group chat reading all the other messages at once and updating what they mean based on the full conversation — not waiting their turn. That parallel, everyone-reads-everyone step is self-attention.

How it works

For a sentence of n tokens, self-attention builds an n × n grid of relevance:

Each token produces a query, key, and value.
Every token's query is compared with every token's key → an n × n attention matrix.
Softmax turns each row into weights that sum to 1.
Each token's output = weighted sum of all tokens' values.

Because all comparisons happen together, self-attention is fully parallel — a big reason transformers train so efficiently.

For each token i:
  look at all tokens j (including itself)
  weight_ij = how relevant token j is to token i
  new_i = Σ (weight_ij × value_j)

Real-world example

In "The trophy didn't fit in the suitcase because it was too big," self-attention lets "it" gather strong context from "trophy." Swap to "…too small," and "it" leans toward "suitcase." The same self-attention mechanism handles both — it learned the relationships from data.

Technical explanation

Self-attention means Q, K, V are all projections of the same input x:

Q = xWᵠ,  K = xWᴷ,  V = xWⱽ
Attention(Q,K,V) = softmax(QKᵀ / √dₖ) V

In decoder (generative) transformers, a causal mask prevents a token from attending to future tokens — position i may only see positions ≤ i, so the model can't "cheat" while predicting the next word. Cost scales as O(n²) with sequence length n, which is why long context windows are expensive and motivate the KV cache and efficient-attention variants.

Common mistakes

Common mistake

Assuming a token can't attend to itself. It can — and often should. A word's own value is part of the weighted blend.

Forgetting the causal mask in generative models — without it, the model would peek at future words.
Overlooking the O(n²) cost, which grows quickly with longer inputs.

When to use it

The default context-mixing step inside every transformer block.

When NOT to use it

Extremely long sequences on tight budgets, where sparse/linear-attention variants are preferred.

Alternatives

Sparse, sliding-window, or linear attention reduce the O(n²) cost while approximating full self-attention.

Quick quiz

Related concepts

What is Attention? — Attention lets each word decide which other words matter most, so a model can understand context.
What is Multi-Head Attention? — Multi-head attention runs several attention computations in parallel, each learning to focus on a different kind of relationship.
What is Positional Encoding? — Positional encoding adds word-order information to a transformer, because attention alone treats input as an unordered set.