Context Window

What Is Context Window?

A context window is the maximum amount of text an AI model can process in a single request, measured in tokens, spanning the system prompt, conversation history, retrieved documents, tool results, and the model's own output. It functions as the model's working memory: anything inside it can influence the answer, and anything outside it does not exist for the model.

Key Takeaways

• Everything shares one budget. Instructions, history, files, tool output, and the generated response all draw from the same token pool, so a long transcript directly crowds out room to think.
• Windows grew from a few thousand tokens to a million-plus in a few years, which changed what is possible without changing what is wise. Anthropic's expansion of Claude Sonnet 4 to 1 million tokens in August 2025 was a 5x jump, enough to fit a codebase of over 75,000 lines in a single request [1].
• Advertised size and effective size differ. Recall and reasoning quality degrade over very long inputs, so the last hundred thousand tokens are rarely as useful as the first.
• The window has no persistence. When a session ends or history is trimmed, that information is gone unless something like agent memory deliberately writes it down.

How It Works

Models read text as tokens, chunks of roughly three to four English characters, and the architecture fixes a maximum number the model can attend to in one request. That ceiling keeps moving: Google launched Gemini 1.5 Pro in 2024 with a standard 128,000-token window and reported successfully testing up to 10 million tokens in research [2]. Within that limit, the attention mechanism lets every token relate to every other, which is what allows an answer on the final line to draw on a detail from the first. That same mechanism explains the ceiling: attention cost grows steeply with length, so serving long windows takes real memory and compute, which providers claw back with tricks like caching the unchanged prefix of repeated requests. Practical consequences follow directly, including why long inputs cost more and why keeping the stable part of a prompt identical across calls is cheaper than shuffling it.

The window is a rolling constraint rather than a one-time check. In a conversation or agent session, each request repackages the accumulated history plus the new turn, and once the total approaches the limit something must give: old turns get dropped, summarized through compaction, or offloaded to external storage and retrieved on demand via RAG-style lookup. This is the mechanical reality underneath context engineering, the discipline of spending the budget well.

Capacity and quality also part ways as inputs grow. A model with a million-token window will accept a million tokens, but retrieval of specific details from the middle of enormous inputs is measurably less reliable than from the edges, and irrelevant bulk actively distracts, the degradation pattern called context rot. The gap is large: the 2025 NoLiMa benchmark tested 13 LLMs claiming 128K-plus windows and found GPT-4o dropped from a 99.3% short-context baseline to 69.7% at just 32K tokens [3]. Practitioners therefore reason about an effective window, the length at which the model still performs near its best, and treat headroom beyond it as space for emergencies rather than a target to fill.

Example

A coding agent starts a large migration in a 200,000-token window. Session start is cheap: instructions and the task brief use 6,000 tokens. Three hours in, the history holds dozens of file reads, test runs, and diffs, and the window is at 85 percent. Symptoms appear before any hard error: the agent re-reads files it already saw and re-litigates a design decision from hour one, because the record of that decision was buried two hundred pages back. The harness compacts, replacing the oldest transcript with a summary of decisions made and files touched, freeing 60 percent of the budget. The agent sharpens immediately, and the team adds a rule to compact at 60 percent instead of waiting, plus a scratchpad file where the agent records key decisions so nothing load-bearing lives only in the transcript.

What People Get Wrong

The common error is treating a giant window as a solution to memory, on the logic that a million tokens means the whole codebase can just be pasted in. It fails twice. Within a session, stuffing the window degrades the model's precision on the parts that matter while multiplying cost, so the dump is worse than a curated selection of relevant files. Across sessions, the window solves nothing at all, because it is working memory, not storage: tomorrow's session starts empty regardless of size. Durable knowledge needs an actual mechanism, instruction files, memory systems, retrieval, and a bigger window changes none of that. Window size is capacity; memory is architecture.

FAQ

What happens when a context window fills up? The request either gets rejected for exceeding the limit or, in managed chat and agent products, older content is silently dropped or summarized to make room. That trimming is why long sessions "forget" early details: the tokens are genuinely no longer in the model's view.

Is a bigger context window always better? Bigger is more capable but not automatically better used. Long inputs cost more per request, slow responses, and dilute attention, and models retrieve less reliably from the middle of huge contexts. A relevant 20,000 tokens routinely beats an indiscriminate 500,000.

How is a context window different from memory? The window is volatile working memory inside one request cycle; it persists nothing. Memory in agent systems means explicit machinery, files, databases, retrieval, that writes information down outside the window and reloads it later. Confusing the two is how teams end up surprised that their agent remembers nothing tomorrow.

Sources

1. Anthropic. "Claude Sonnet 4 context window expanded to 1 million tokens, a 5x increase covering codebases over 75,000 lines." https://claude.com/blog/1m-context. Accessed August 2026.
2. Google (official blog). "Gemini 1.5 Pro launched with a 128,000-token standard context window; up to 10 million tokens tested in research." https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/. Accessed August 2026.
3. Modarressi et al., arXiv (NoLiMa). "13 LLMs with 128K+ claimed windows tested; GPT-4o fell from 99.3% short-context baseline to 69.7% at 32K tokens." https://arxiv.org/abs/2502.05167. Accessed August 2026.