Intermediate8 min read

What is Attention?

Attention is how each word looks at all the other words and decides which ones matter for understanding it.

Explain like I'm new to AI

Read this sentence:

"The animal didn't cross the street because it was too tired."

What does "it" mean — the animal or the street? You know instantly it's the animal. How? You paid attention to the right earlier word.

Attention gives a model the same ability. For every word, it computes how much to "look at" every other word, then blends in the most relevant context. That's what lets models resolve pronouns, track topics, and understand meaning.

Try it — click any word to see what it attends to:

Click any word to see what it pays attention to. Darker = stronger attention.

Query word: it — notice "it" attends most strongly to "animal", resolving what "it" refers to.

Attention lets each word gather context from the most relevant other words.

Mental model

Think of attention as spotlights. When processing a word, the model shines brighter spotlights on the words that help explain it, and dimmer ones on the rest. The word's new meaning is a blend of everything it lit up.

How it works

Attention is built from three roles for each token, all learned:

Query (Q): "What am I looking for?"
Key (K): "What do I offer?"
Value (V): "What information do I carry?"

The steps:

For a given word's query, compare it against every word's key to get a score (how relevant each word is).
Turn scores into weights that add up to 1 (via softmax).
Take a weighted sum of the values using those weights.
The result is the word's context-aware representation.

scores   = Q · Kᵀ            # relevance of every word to this one
weights  = softmax(scores)   # normalize to add up to 1
output   = weights · V       # blend the most relevant information

Real-world example

In "I poured water from the jug into the cup until it was full," attention helps "it" attend to "cup" (the thing being filled) rather than "jug." Change one word — "until it was empty" — and a well-trained model shifts attention toward "jug." Same words, different focus.

Technical explanation

This is scaled dot-product attention: softmax(QKᵀ / √dₖ) · V, where dividing by √dₖ keeps scores from growing too large. When Q, K, and V all come from the same sequence, it's self-attention (the core of transformers). When queries come from one sequence and keys/values from another, it's cross-attention (used in encoder-decoder models). Running several attention computations in parallel gives multi-head attention.

Common mistakes

Common mistake

Thinking attention weights are a full explanation of the model's reasoning. They show where the model looked, which is insightful but not a complete account of why it answered as it did.

Assuming the highest-attention word is always the "answer" — attention blends many words.
Confusing attention (the mechanism) with self-attention (attention applied within one sequence).

When to use it

Any time context between elements matters — language, and increasingly vision and audio.

When NOT to use it

Purely local, fixed-window problems where a convolution is cheaper, or extremely long inputs without efficient-attention variants.

Alternatives

Recurrence (RNNs) and convolutions capture context differently, but attention is the modern default.

Quick quiz

Related concepts

What is Self-Attention? — Self-attention is attention applied within a single sequence, letting every word gather context from every other word in the same text.
What is Multi-Head Attention? — Multi-head attention runs several attention computations in parallel, each learning to focus on a different kind of relationship.
What is a Transformer? — A transformer is the neural network architecture behind modern AI, using attention to process all words at once and learn how they relate.