User Memory vs Agent Memory
User memory is about the person (preferences, profile, history they own). Agent memory is about how the system works (playbooks, this run's todos). Keep them in different scopes so privacy and behavior don't collide.
Explain like I'm new to AI
Two questions:
- "What do we know about Ada?" → User memory
- "How should this agent book lunch or file a ticket?" → Agent memory
About the person. Scoped to that user. They should be able to see, edit, and delete it.
- • Diet, timezone, pronouns
- • "Don't book mornings"
- • Past orders and tickets
Leakage here is a privacy incident.
About how this agent works: playbooks, tool habits, this run's checklist. Not the user's identity.
- • "Always confirm before refunds"
- • Learned query pattern for this DB
- • Current task todos
Sharing this across users is often fine; sharing user memory is not.
If you dump both into one "memories" index, two disasters appear: Ada's medical note gets retrieved for Bob, and a playbook line ("always refund") gets treated as Ada's personal policy.
Mental model
A hospital. The patient chart is user memory (access-controlled, erasable). The nursing protocol binder is agent/procedural memory (shared by staff, versioned). You would never file protocols inside a patient's chart — or charts in the protocol binder.
How it works
User memory
- Scoped by
user_id/ tenant on every write and retrieve - Visible in a UI: list, edit, delete (GDPR-style erasure)
- High sensitivity: minimize, encrypt, audit
- Semantic (preferences) + episodic (their past tickets)
Agent memory
- Scoped by
agent_id/ environment, not by end-user when it's a shared skill - Procedural playbooks, tool-use patterns, org-level constraints
- Run-level state (todos) is agent memory that should die with the job, not land in Ada's profile
A coding assistant's AGENTS.md is agent/procedural memory. "Ada prefers tabs" is user memory. "Current PR checklist" is run state.
Real-world example
A travel agent stores Ada.diet = vegetarian (user, semantic). It also has a procedure "never book without checking visa" (agent, procedural). When Bob chats, you load Bob's user memory + the shared procedure — never Ada's diet.
Technical explanation
Authorization is part of the retrieve path: metadata filters (user_id = current_user) are security controls. Hybrid search without filters is a data leak.
Multi-agent systems need the same split: a researcher agent may share procedural memory with a writer, but must not share user memory across customers. Isolated stores beat "one big brain."
Product-wise, show users their stored facts. Silent user memory destroys trust and is a compliance problem (right to access/erasure). Agent memory can stay internal, but still needs versioning.
Common mistakes
One vector namespace called "memory" for everyone and every agent. You will eventually retrieve the wrong person's life into the wrong prompt.
- Writing run todos into the user's long-term profile ("Ada is currently debugging auth.ts").
- Training users that "the AI just remembers" with no way to inspect or wipe.
When to use it
- Always, as soon as you persist anything about a person or share an agent across users.
When NOT to use it
- A single-user local tool with no persistence beyond the session — still keep run state out of "profile" if you add memory later.
Alternatives
- Fetch profile from the real CRM/IdP with a tool instead of duplicating it in a memory DB (source of truth wins).
Quick quiz
Related concepts
- Long-term Memory — Long-term memory stores distilled facts outside the model so a new session can retrieve them — the weights never learned the fact.
- Agent Memory & State — Agents use short-term memory (the context window), long-term memory (facts stored outside and retrieved), and state (task progress) to stay coherent over many steps.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI