Context is what the model sees in the current inference call; memory is what persists across inferences.
The distinction is the most fundamental trade-off in agent design. Context is bounded (the model's context window is fixed, typically 8K-200K tokens); memory is unbounded (the agent can store as much as the storage layer allows). The agent's job is to bridge the two: the agent decides what to put in the context, and the agent decides what to persist to memory.
The three layers
In practice, the agent's state is in three layers:
1. Context. The current inference call's input. The context is smallest, fastest, and most expensive. The context is what the model sees.
2. Working memory. The session's recent history. The working memory is larger than the context, slower to access, and cheaper per token. The working memory is what gets assembled into the context.
3. Long-term memory. The persistent store across sessions. The long-term memory is larger than the working memory, slower to access, and cheapest per token. The long-term memory is what survives across sessions.
The trade-off
The context vs. memory trade-off is the most expensive decision the agent makes. The wrong choice has three failure modes:
1. Too much context. The agent puts too much in the context. The model is distracted by irrelevant information, the cost per inference is high, and the latency is high. The agent is "over-eager."
2. Too little context. The agent puts too little in the context. The model is missing information it needs to make the decision, the agent re-asks the user, and the work is slowed. The agent is "under-informed."
3. Wrong memory. The agent writes the wrong things to memory. The next session gets stale information, the agent's behavior is inconsistent, and the operator's trust is eroded. The memory is "noisy."
The right pattern
The right pattern is to keep the context minimal and the memory maximal. The agent assembles the context for the current inference call from the memory, and the agent writes the inference's outcome back to the memory. The agent's compaction step is what keeps the context minimal: the compactor summarizes the past, and the summary is what goes into the context.
The right pattern also means that the memory is structured. The memory is not just a blob of text; the memory is indexed, the memory is searchable, and the memory is queryable. The embedding index and the graph-based memory index are the typical implementations.
Operator implications
The context vs. memory decision is the right place to start when an operator is debugging an agent that is producing wrong answers or inconsistent behavior. The most common operator issues are: the context is too small (the agent is missing information), the memory is too noisy (the agent's retrieval is returning irrelevant results), the memory is too slow (the agent is timing out before the retrieval is complete), and the memory is too sparse (the agent is not writing the right things to memory).
The context vs. memory decision is also the right place to start when an operator is adding capabilities. The canonical question is "what should the agent remember, and what should the agent ignore?"
Related terms
Context and memory are the two layers of the agent loop's state. The boundary between context and memory is managed by compaction. The memory is implemented as embedding search or graph-based memory. The memory's quality is one of the inputs to the observability layer.
For the full primer, see First Steps: Sessions and Memory.