Intermediate7 min read

What is Multi-Head Attention?

Multi-head attention runs several attention "heads" in parallel, each learning to focus on a different kind of relationship, then combines them.

Explain like I'm new to AI

One attention calculation can only focus on so much at once. So transformers run several in parallel — called heads — and let each specialize.

One head might learn to link subjects to verbs, another to track pronouns ("it" → "animal"), another to watch nearby phrasing, and another to follow the overall topic. Combine all their findings and each word ends up with a much richer understanding.

Same words in

Head 1

Who did what (subject → verb)

Head 2

What refers to what (pronouns)

Head 3

Nearby words / phrasing

Head 4

Long-range topic links

One richer representation out

Multiple heads let the model attend to several kinds of relationships at once.

Mental model

Imagine editing an essay with a panel of specialists: a grammar checker, a pronoun-tracker, a tone reader, and a fact-linker — all reading at the same time. Each notices different things; you merge their notes into one improved draft. That panel is multi-head attention.

How it works

Split each token's vector into h smaller pieces (one per head).
Run self-attention independently in each head — each with its own learned Q/K/V projections, so each head can focus differently.
Concatenate the heads' outputs back together.
Pass through a final linear layer to combine them into one vector.

Each head works in a smaller space, so running many heads costs about the same as one big attention — but captures far more.

headᵢ = Attention(xWᵠᵢ, xWᴷᵢ, xWⱽᵢ)
MultiHead(x) = Concat(head₁, …, head_h) · Wᴼ

Real-world example

In "The lawyer told the client that she would win the case," different heads help resolve who "she" is by weighing "lawyer" vs "client," while other heads track the verb "win" and the object "case." The combination yields a coherent reading.

Technical explanation

With model dimension d_model and h heads, each head uses dimension d_k = d_model / h. All heads run in parallel; the concatenated result (size d_model) is projected by Wᴼ. Modern LLMs often use efficiency variants: Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) share keys/values across heads to shrink the KV cache and speed up inference with minimal quality loss.

Common mistakes

Common mistake

Thinking more heads always means better results. Beyond a point, extra heads add cost with little gain — and some heads end up redundant. It's a balance, not "more is always better."

Assuming humans assign each head a job — specializations emerge from training, they aren't hand-coded.
Confusing multi-head attention with stacking layers; heads are parallel within a layer, layers are sequential.

When to use it

Always, in practice — it's a standard part of every transformer block.

When NOT to use it

Only when using deliberate efficiency variants (MQA/GQA) to reduce memory at scale.

Alternatives

MQA and GQA are optimized forms of multi-head attention for faster, cheaper inference.

Quick quiz

Related concepts

What is Self-Attention? — Self-attention is attention applied within a single sequence, letting every word gather context from every other word in the same text.
What is Attention? — Attention lets each word decide which other words matter most, so a model can understand context.