What is a Token?
A token is the small chunk of text — usually a word or piece of a word — that an LLM reads and generates; models think in tokens, not letters or words.
Explain like I'm new to AI
An LLM doesn't see text the way you do. It doesn't read letters, and it doesn't quite read whole words either. It reads tokens — small chunks that are often a short word, part of a longer word, or a piece of punctuation.
A rough rule of thumb in English: 1 token ≈ 4 characters ≈ ¾ of a word. So "cat" is one token, but "unbelievable" might be split into "un", "bel", "iev", "able".
Why care? Because tokens are the unit of everything: how much text fits in the model (context window), how fast it responds, and how much you pay.
Mental model
Think of LEGO bricks. You don't describe a castle brick-color by brick-color; you build it from standard bricks. Tokens are the model's standard bricks of language — every input and output is assembled from them.
How it works
- Your text is broken into tokens before the model sees it.
- Each token maps to a number (an ID) the model can process.
- The model predicts the next token, over and over, to generate text.
- Common words are usually one token; rare or long words split into several.
| You see | The model sees | |
|---|---|---|
| Unit | Letters and words | Tokens (word-pieces) |
| Example | "I love astronomy" | ["I", " love", " astro", "nomy"] |
| Count | 3 words | ≈ 4 tokens |
Real-world example
API pricing is per token. If a prompt is 1,000 tokens and the reply is 500 tokens, you're billed for ~1,500 tokens. Emoji, code, and non-English text often use more tokens per character — which is why the same paragraph can cost more in one language than another.
Technical explanation
Modern LLMs use subword tokenization (e.g. Byte-Pair Encoding). A fixed vocabulary (often 50k–200k+ entries) is learned so that frequent character sequences become single tokens while rare ones split into pieces. This balances a small vocabulary against short sequences. Every token maps to an embedding vector; the model's final layer outputs a probability over the entire vocabulary for the next token.
Common mistakes
Assuming one word equals one token. Long or rare words split into several tokens, and a leading space is often part of the token (" cat" vs "cat"), which can surprise you when counting.
- Counting characters or words to estimate cost — count tokens instead.
- Forgetting that both the prompt and the response consume tokens.
When to use it
- Any time you budget for context, latency, or cost — think in tokens.
When NOT to use it
- Not applicable — tokens are fundamental to how LLMs work.
Alternatives
- Character-level and word-level models exist, but subword tokens are the standard sweet spot.
Quick quiz
Related concepts
- What is Tokenization? — Tokenization is the process of splitting text into tokens using a learned subword vocabulary before a model can read it.
- What is a Context Window? — The context window is the maximum number of tokens an LLM can consider at once — its working memory for a single request.
Last reviewed: 2026-08-30 · Written by ByHeart AI · Reviewed by ByHeart AI