Why Transformers Replaced RNNs

Transformers replaced RNNs for language because every token can see every other token in one hop and the whole sequence trains in parallel. The recurrent loop was the bottleneck; attention deleted it.

Explain like I'm new to AI

You have the pieces. RNNs / LSTMs read left to right, stuffing history into a state. Transformers (category next door — read What is a Transformer? if the block diagram is new) throw away the loop: self-attention builds an n×n "who matters to whom" grid and mixes values in one shot, then an MLP (this category's first lesson) refines each position.

This lesson is the why, not a re-teach of Q/K/V. Two engineering facts decided 2017–2026:

1. Parallelism. GPU/TPU silicon wants big matmuls. An RNN's time axis is a for-loop: h_t waits on h_{t-1}. Teacher-forced training of a transformer is one parallel pass over the sentence (with a causal mask in decoders so you don't peek). Wall-clock on a corpus is not even close.
2. Path length. In an RNN, "The" influencing "$120" at the end of a sentence is T−1 recurrent hops, each a chance to vanish. In attention, it is one weighted lookup (then however many blocks you stack — still O(1) in sequence length for the graph distance). Long-range agreement ("the keys … are") stopped being a research crisis.

LSTMs were a brilliant patch on (2) and did nothing for (1). Once attention worked, RNNs lost the plot for NLP. This chain's last hop is self-attention — the mechanism in detail.

Mental model

RNN: a single-file hallway. Transformer: a meeting where everyone can pass a note to everyone at once. Training a corpus is "run 10,000 meetings in parallel on a GPU." Training LSTMs is "10,000 hallways, walk them step by step."

The remaining RNN hallway: true streaming with a tiny state and no budget to store K/V for every past frame. That hallway got narrower every year (chunked attention, SSMs, linearized attention) but it did not vanish.

How it works

Serial vs parallel (training). Language modeling with teacher forcing:

RNN:  for t in 1..T:  h_t = RNNCell(h_{t-1}, x_t);  loss += CE(y_t, x_{t+1})
XF:   H = Transformer(x_1..x_T, causal_mask);      loss = CE(H, x_{2..T+1})  # one graph

The transformer still generates token-by-token at inference (decode). So do RNNs. The difference that scaled labs is training and encoding a full context without a Python-time loop over T.

Information highway. Click $120 in the visual: RNN path is The → refund → cap → is → $120. Each arrow is W_h and a nonlinearity. Transformer path is a cell in the attention matrix. Stacking 12 blocks is 12 such meetings, not 12×T hops of a chain.

Cost trade. Attention is O(T²) in sequence length (every pair). RNNs are O(T) compute but O(T) serial depth. For T=512, 2017 hardware preferred T² that vectorizes over T that doesn't. For T=100k, T² hurts — hence KV cache, flash attention, sliding windows, SSMs. The replacement was not "free infinite context"; it was "better constant-depth mixing + parallelism" in the regime that mattered.

What about CNNs on text? 1-d convs are parallel and local. They never got long-range without huge dilation stacks. Transformers got long-range "for free" (until T²). That's why conv language models stayed a footnote and conv vision did not die.

Real-world example

GNMT (2016): 8 LSTM layers, days of training, still awkward on long sentences. Transformer NMT (2017): same BLEU with a fraction of the train time, then a year later everyone copied the block into GPT/BERT. By 2020 "sequence model" in a job description meant transformer unless you said otherwise. 2026: an engineer proposing a 4-layer LSTM language model needs a specific constraint (microcontroller, 20-d state, no attention SRAM), not nostalgia.

Technical explanation

Vaswani et al. literally titled the paper Attention Is All You Need — no recurrence, no convolution in the mixer. Positional encodings put order back (RNNs had order for free via the loop). Residual + LayerNorm made depth easy; LSTMs needed tricks to stack past 4–8.

Inductive bias given away: recurrence forces a Markov-like state. Attention can ignore order except via positions, and can overfit pairwise patterns. We accepted that because data + scale + positional structure were enough. When data is tiny and the process is actually a dynamical system, an RNN or SSM prior can still be the right bias.

SSMs / Mamba and linearized attention are 2023–2026 answers to "can we have O(T) and long memory and still scan in parallel (via associative scans)?" They compete with transformers on long 1-d streams; they have not replaced transformers for general NLP the way transformers replaced LSTMs. Mention them as the living RNN-adjacent research line, not as a rewrite of history.

Decode-time. Autoregressive transformers are sequential in output length, which is why KV cache exists. That is not the RNN training bottleneck coming back. Do not confuse "generation is a loop" with "the architecture is recurrent."

Common mistakes

• Claiming RNNs are extinct. Streaming, tiny time series, some speech front-ends, and teaching still use them.
• Thinking inference is fully parallel for a GPT-style model — prefill is; decode is a loop (inference category).
• Assuming O(T²) means transformers lost for long context. Engineering (sparse, linear, SSM hybrids, 2026 long-context models) is the continuation, not a return to vanilla LSTMs as the default.

When to use it

• This comparison: whenever you choose a sequence backbone. Default transformer for language and a lot of audio/vision.
• As the bridge into the Transformers category (next: self-attention).

When NOT to use it

• Don't reach for a vanilla RNN "to be efficient" on a 4k-token NLP task — you will be inefficient on a GPU and worse on long-range.
• Don't use a full transformer on a 32-step, 8-d sensor if a GRU hits the SLA; the inductive bias and RAM budget can still favor a state.

Alternatives

• Gated RNNs when the state must be tiny and streaming.
• 1-d CNNs / TCNs for local temporal patterns.
• SSMs / hybrid architectures for very long 1-d sequences — watch this space; not a reason to skip learning attention.

Quick quiz

Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI