Agent Memory & State
Agents combine short-term memory (what's in the context window right now), long-term memory (facts stored outside the model and retrieved when relevant), and state (the task's plan and progress) to act coherently across many steps.
Explain like I'm new to AI
An agent loops for many steps — but the model itself has no memory between calls. Everything it "remembers" has to be put back into the context each time. Agents manage three kinds of memory:
The current context window: the task, recent steps, and tool results. Fast but limited and temporary.
- • This conversation
- • Scratchpad of recent actions
- • Latest tool outputs
Facts stored outside the model (often a vector database) and retrieved when relevant. Survives across sessions.
- • User preferences
- • Past conversations
- • Learned facts
The structured progress of the task: the plan, what's done, and what's left. Keeps a long task on track.
- • Current plan / to-do list
- • Completed sub-tasks
- • Variables & results
- Short-term = what's in the context window now (limited, temporary).
- Long-term = facts saved outside the model and fetched when needed (survives sessions).
- State = the structured plan and progress that keeps a long task on track.
Mental model
Like a worker with a notepad (short-term: today's scratch notes), a filing cabinet (long-term: retrieve old records when relevant), and a checklist (state: what's done, what's next). The notepad clears often; the cabinet persists; the checklist tracks progress.
How it works
- Short-term: recent steps and tool outputs live in the context window. It's fast but finite, so long loops must summarize or drop old content.
- Long-term: important facts are written to a store — often a vector database — and retrieved (RAG-style) when relevant to the current step.
- State: an explicit structure (plan, to-do list, variables) the agent reads and updates, so a 30-step task doesn't lose the thread.
Real-world example
A personal assistant agent remembers your preferences ("I'm vegetarian") across weeks — that's long-term memory in a store, retrieved when booking dinner. Within one booking task, the current conversation is short-term, and the checklist ("find restaurant → check hours → reserve") is state.
Technical explanation
Because context is limited and expensive, memory is an engineering problem:
- Context management: summarize old turns, keep only relevant snippets, and use the context window deliberately (this is a big part of context engineering).
- Long-term stores: embeddings + vector search for semantic recall; sometimes structured databases or knowledge graphs for facts.
- Memory writing policy: deciding what to persist (not everything) and when to retrieve it.
- State persistence: frameworks checkpoint state so an agent can pause, resume, or recover after failure.
Getting memory right is often what separates a demo from a reliable agent — poor memory shows up as forgetting instructions, repeating work, or contradicting earlier steps.
Common mistakes
Assuming the model "remembers" across steps. It doesn't — anything not placed back into the context is gone. Long-running agents must actively manage what to keep, summarize, or retrieve.
- Dumping everything into context until it overflows (costly and degrades quality).
- Persisting too much noise to long-term memory, then retrieving irrelevant junk.
When to use it
- Any multi-step or multi-session agent — memory and state are what make continuity possible.
When NOT to use it
- Single-shot tasks that fit in one context need no long-term store or state machinery.
Alternatives
- For pure fact recall, plain RAG may be enough without full agent state management.
Quick quiz
Related concepts
- The Agent Loop — The agent loop is the cycle of observe, reason, plan, and act that repeats — using tools and results — until the agent reaches its goal.
- What are Embeddings? — Embeddings turn information into numerical vectors where similar meanings sit close together, so software can compare meaning with math.
- State Management — State is the structured progress you persist outside the raw transcript — using write, select, compress, and isolate so the next call gets a clean working set.
- Short-term Memory & Conversation History — Short-term memory is two layers — the session log you store and the smaller working set that actually fits in the context window.
- Long-term Memory — Long-term memory stores distilled facts outside the model so a new session can retrieve them — the weights never learned the fact.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI