Agent Memory

What Is Agent Memory?

Agent memory is the set of mechanisms that let an AI agent retain and reuse information beyond a single context window: session state, notes written to files, databases, and vector stores. It is what allows an agent to carry facts, preferences, and lessons across tasks instead of starting every session from zero. The gap it fills is well documented: on the LoCoMo benchmark of very long conversations, averaging 300 turns across up to 35 sessions, Maharana and colleagues found in 2024 that LLM agents substantially lag human performance at recalling earlier context [1].

Key Takeaways

• Models are stateless. Everything an agent "remembers" is engineered around the model, by putting the right stored information back into context at the right time.
• The working split is short-term versus long-term: the current context window and session scratch state on one side, durable stores that survive the session on the other.
• Plain files are a legitimate memory system. Agents that write and reread markdown notes, an approach used by CLAUDE.md-style project files, often beat elaborate vector setups on reliability and debuggability.
• Retrieval is the hard half. Storing everything is easy; surfacing the three relevant memories at the right moment, without flooding the context, is the actual engineering problem.

How It Works

Short-term memory is the context window itself: the conversation, tool outputs, and files read this session. It is fast and complete but finite, and it evaporates when the session ends. Agents extend it within a session through compaction, summarizing older turns to reclaim space, and through delegation, where a subagent absorbs a noisy task and returns only the conclusion.

Long-term memory persists outside the model. The common substrates are files the agent reads and writes, structured stores such as SQL or key-value records for facts and preferences, and a vector database for semantic recall over large histories, retrieved on demand in the style of retrieval-augmented generation. Purpose-built layers of this kind measurably outperform bolted-on features; the Mem0 memory layer scored a 26% relative improvement over OpenAI's memory feature on the LOCOMO benchmark in its 2025 evaluation [2]. Around these substrates sit three processes: deciding what is worth writing (salience), fetching what is relevant to the current task (retrieval), and revising or discarding entries that have gone stale (maintenance). Skipping maintenance is how memory rots: an agent that recorded "deploys happen from the release branch" keeps acting on it long after the team switched to trunk-based deploys.

In multi-agent settings, memory also serves coordination. A shared store lets one agent's findings inform another's work, and it becomes the system's institutional knowledge. That power carries risk, because a memory poisoned once, whether through error or prompt injection buried in processed content, propagates to every future session that retrieves it, so writes to long-term memory deserve validation and provenance tracking.

Example

A team runs a coding agent on their monorepo daily. In week one, every session rediscovers the same facts: which package manager, how to run the affected-only test suite, the flaky integration test to ignore. They add a memory file the agent maintains, seeded through their project's CLAUDE.md conventions. Now, when the agent learns that the payments tests need a local stub server, it appends that to the memory file with a date. The next session reads the file at startup and skips forty minutes of rediscovery. Once a month an engineer prunes the file, deleting entries invalidated by refactors. The whole system is a few hundred lines of markdown under version control, reviewable in pull requests like any other change.

What People Get Wrong

The reflex is to equate agent memory with a vector database and start there. Semantic search over embeddings is one retrieval technique, useful when memories are numerous and loosely structured, but it is probabilistic: it surfaces what is similar, resurfaces what is outdated, and misses what is phrased differently. Facts that must be reliably present every session, such as conventions, credentials locations, and hard constraints, belong in deterministic memory: a file or record loaded on every run. Most teams need the boring layer first and the semantic layer only after the boring layer overflows.

Long-term memory

Long-term memory is the durable tier, and it is conventionally split three ways, borrowing loosely from cognitive science. Episodic memory records what happened: past sessions, decisions, and outcomes, useful for "have we tried this before?" Semantic memory stores facts and preferences: the user's stack, the project's conventions, the customer's plan tier. Procedural memory captures how to do things, encoded as learned instructions or updates the agent writes into its own guidance files. The tiers differ in how they are best stored: episodic history suits append-only logs with semantic search over them, semantic facts suit structured records with explicit updates, and procedural knowledge suits versioned instruction files that humans can review. What makes any of them "long-term" is survival across sessions plus a retrieval path back into context, because a memory that never gets retrieved might as well not exist.

FAQ

Do LLMs have built-in memory? No. A large language model retains nothing between API calls; its weights are frozen and its context is discarded. Product features marketed as memory, in ChatGPT or Claude alike, are engineered layers that store information externally and reinsert it into future prompts.

What is the difference between agent memory and the context window? The context window is what the model can see right now, and it is the only place memory becomes usable. Agent memory is the machinery that decides what earns a place in that window: what gets saved when the window is full or the session ends, and what gets loaded back later. That selectivity is also an economics decision: Mem0's 2025 benchmarks showed a dedicated memory layer delivering 91% lower p95 latency and saving more than 90% of token cost compared with feeding the full conversation history into context [3].

How do you keep agent memory from degrading over time? Treat it like data with a lifecycle. Record provenance and dates on entries, prefer updating a fact over appending a contradiction, prune on a schedule, and keep human-reviewable memory (files in version control) for anything that steers agent behavior.

Sources

1. Maharana et al. (arXiv). "LoCoMo benchmark: LLM agents lag humans at recalling context in very long conversations averaging 300 turns." https://arxiv.org/abs/2402.17753. Accessed August 2026.
2. Mem0 (arXiv). "Mem0 achieves 26% relative improvement over OpenAI's memory feature on the LOCOMO benchmark." https://arxiv.org/abs/2504.19413. Accessed August 2026.
3. Mem0 (arXiv). "Dedicated memory layer delivers 91% lower p95 latency and over 90% token cost savings versus full conversation history." https://arxiv.org/abs/2504.19413. Accessed August 2026.