What Is a Context Window? Why Does AI Forget?

A context window is the maximum amount of text an AI model can hold in its “memory” at once; it’s measured in tokens. Anything beyond that limit falls out of the model’s reach. This is why, in a long conversation, a model behaves as if it has “forgotten” what was said at the very beginning — it hasn’t actually forgotten, that information has simply fallen outside its window. For a business owner, the practical meaning is this: the longer you talk with AI, the lower the chance it remembers older details.

How a Context Window Works — A Simple Analogy

I think of this like a table. Only so many sheets of paper fit on the table. When a new sheet arrives (a new message from you, a piece of text you paste, the model’s own reply) and the table is full, the oldest sheet falls off the back — there’s no other way to make room.

A context window works exactly like this. Every word is broken into pieces called “tokens” (a word in English is usually about one token), and the window counts these tokens. When the window is full, the model doesn’t grow its capacity — it either refuses to accept new input, or — in an ongoing conversation — the oldest parts of the conversation fall outside the area the model can “see” while producing its next reply. Like the paper on the table: the content still exists somewhere, but the model can no longer look at it.

Why It Matters — Practical Consequences

This isn’t an abstract technical detail, it’s a limit you’ll run into directly in everyday use. Three concrete scenarios:

A 100-page contract: When you paste a long legal document all at once, it might not fit in the model’s window. Either the document gets cut off (the part the model sees stays incomplete), or the tool forces you to split it into pieces.
An hours-long session: In a long working session (say, a back-and-forth conversation over a project), a rule or constraint you set at the start — “always answer in Turkish,” “the budget shouldn’t exceed $5,000” — can silently fall out of the window. The model starts behaving as if it never heard it; it’s not acting in bad faith, it genuinely can no longer access it.
Re-stating critical information helps: Rewriting an important constraint or decision toward the end of a long conversation isn’t paranoia, it’s a practical precaution. You move the information to the “new” end of the window, so the model can genuinely access it in its next reply.

The Difference Between Models

Not every model ships with the same size window. Some models are limited to a window of a few thousand tokens, while others can process hundreds of thousands (or more) tokens at once. A large context window is a real advantage when you want to analyze a long document (a full contract, a technical manual, a codebase) all at once without splitting it into pieces.

But this capacity has a cost: as the number of tokens processed in a request grows, the cost of that request usually grows too. So “the model with the biggest window” isn’t automatically the right economic choice for every job — there’s no point carrying a giant window (and its cost) for a short question. I explain how tokens are priced and how that translates into cost in more detail on the What Is a Token: The Cost of Using AI page.

Ways to Work Around the Context Window Problem

You can’t remove the limit, but there are a few practical ways to work around it:

Split the document into pieces and feed it in order. Instead of processing a 100-page text all at once, work through it in logical sections; produce a short summary after each section and give it to the model along with the next piece.
Summarize important information and put it back at the start and end. Models tend to use information at the beginning and end of a given text more reliably than information in the middle. Reminding the model of a critical constraint both at the start and the end reduces the risk of it getting lost.
Use the RAG (Retrieval-Augmented Generation) approach. Instead of cramming your entire knowledge base into the model, you find just the relevant piece based on the question asked and give that to the model at that moment. This lets you work with a much larger knowledge pool without ever straining the window — I cover this topic on the What Is RAG page.
Use a persistent setup across sessions. Instead of re-explaining the same company documents in every session, you can set up a structure that keeps the model’s view of a certain set of information across sessions — I walk through this step by step on the Setting Up a Company Assistant With Claude Projects page.

An Observation From My Own Experience

I ran into this myself, on one of my clients’ accounts, during a long working session: I was walking through a conversion tracking setup step by step in a conversation that lasted hours. We’d made a clear decision at the start of the session — “GTM won’t be used, it’ll be written directly to gtag.” As the conversation went on and the message count grew, the model started suggesting a GTM-based solution as if that decision had never been discussed. It wasn’t bad faith or a mistake; that decision had simply fallen outside the context window. The fix was simple: I rewrote the constraint in one sentence and we continued. Repeating critical decisions every once in a while during long sessions has since become a standard habit of mine.

You Can Ask Me for This Work

This page is part of the Agentic AI Guide. If you want to set up a document-processing flow that respects the context window limit, or design a long AI-assisted process (conversion tracking, content production, data analysis, an internal company assistant) correctly from the start, you can ask me for this: typically 2 hours to 2 weeks, remote, billed hourly. AI Consulting

Frequently Asked Questions

How many words does a context window hold?

It isn’t measured directly in words, but in tokens. A word in English is usually about one token, so the same token count can correspond to a different word count depending on the structure of the text. A “32,000-token window” might roughly correspond to 24,000-25,000 words of English text — the exact ratio depends on the content.

What exactly happens when the context window fills up?

It depends on the implementation: some interfaces refuse new input and throw an error, others — especially in an ongoing conversation — silently drop the oldest parts of the conversation and continue. In the second case, without you noticing, the model may no longer “see” what was discussed at the start.

Is it possible to increase the context window?

No, the context window is a fixed property of the model you’re using; you can’t increase it. The only things you can do are pick a model with a bigger window, or fit the work into the existing limit using methods like RAG, summarization, or chunking.

Is a bigger context window always better?

For long-document analysis, yes, it’s an advantage. But a bigger window usually comes with a higher cost, and using that capacity for short, simple tasks creates unnecessary expense. The right choice depends on the size of the job.

The Context Window: Why AI 'Forgets'