Short-term Memory & Conversation History
Short-term memory is not "whatever the model feels like remembering" — it is the session you persist plus the much smaller working set that still fits in the context window this call.
Explain like I'm new to AI
People say "the chatbot remembers this conversation." Two different things get mixed up:
- Session / conversation history — every turn saved in a database for this thread. You can reload it tomorrow in this session.
- Working memory — what is actually inside the context window for the next model call. That window is tiny compared with a long thread.
Click through a chat. The left column keeps the whole session. The right column is all the model can see:
- You: I'm Ada. I use a standing desk.
- AI: Noted — standing desk.
- You: Book a 30-min focus block at 2pm.
Everything said this session, on disk.
- You: I'm Ada. I use a standing desk.
- AI: Noted — standing desk.
- You: Book a 30-min focus block at 2pm.
Only the last 4 turns the model can see right now.
When the window fills, older turns are still in the log. They are not in the model's head unless you put them back (summarize, retrieve, or pin).
Mental model
A meeting. The recording is the session log. The notes currently on the whiteboard are working memory. Wipe the board and the recording still exists — but the people in the room only see the board.
How it works
- Append each user/assistant/tool message to a thread-scoped store (Postgres, SQLite, Redis).
- On the next call, select a working set: last N turns, plus a compact summary of the rest (see context compression).
- Working memory also includes the live task state (todos, the current file) — that is context engineering's packing job.
MemGPT-style agents treat this like an OS: RAM = context window, disk = the log and long-term stores, with "paging" when RAM fills.
Real-world example
You tell a support bot your order id in turn 2. In turn 40, if those tokens have fallen out of the window and nobody saved order_id into state, the model will ask again — even though the session log still has it. The bug is selection, not "a forgetful model."
Technical explanation
The weights do not update when you chat. Continuity is 100% what you resubmit. Distinguish:
| Layer | Lifetime | Typical store |
|---|---|---|
| Working memory | One API call | Context window |
| Conversation / session | One thread | DB, often with a summary column |
| Long-term | Across threads | Vector/graph/KV (next lesson) |
Hot path: keep recent turns verbatim (tool errors, the latest user ask). Cold path: compact. Prompt caching likes an append-only recent history so prefixes stay stable.
Common mistakes
Equating "we stored the chat" with "the model remembers." Storage without packing is a log the model cannot see.
- Replaying the entire thread every call (cost, dilution, cache misses when you rewrite it).
- No thread id — messages leak across users or conversations.
When to use it
- Any multi-turn product. Session logs are the default; working-set packing is mandatory once threads get long.
When NOT to use it
- Single-shot APIs with no follow-up — don't invent a history store.
Alternatives
- Stateless turns with all needed facts in one prompt; long-term memory when it must survive a new thread.
Quick quiz
Related concepts
- Agent Memory & State — Agents use short-term memory (the context window), long-term memory (facts stored outside and retrieved), and state (task progress) to stay coherent over many steps.
- Context Compression — When the window fills, trim bulky tool output first, then compact old turns into state — prefer deletion over rewriting, and trigger on a token or turn threshold.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI