Quick answer:
The context window is a language model’s working memory: the maximum amount of text (measured in tokens) it can consider at once. Your question, any documents you paste in, the conversation so far, and the answer being generated all have to fit inside it. Anything outside the window may as well not exist; the model has no other memory.
When people say “the model forgot what I said earlier”, they usually mean the conversation scrolled out of the window.
What is a context window?
A language model is stateless. Every API call starts from zero, and the illusion of an ongoing conversation is maintained by resending the entire history with each message. The context window is the hard cap on that resend: prompt plus history plus attached material plus the space reserved for the reply.
It’s a genuine architectural limit. The transformer attention mechanism compares tokens against each other, and that work grows steeply with length, which is why context was so scarce for so long and why extending it was a real research achievement rather than a config change.
How fast did windows grow?
Fast enough to keep invalidating advice. GPT-3 in 2020 had about 2,000 tokens, a few pages. GPT-4 launched in 2023 with 8,000 and 32,000-token tiers. Anthropic pushed Claude to 100,000 and then 200,000 tokens (a decent novel). Google’s Gemini 1.5 took it to 1 million and then 2 million tokens in preview, entire codebases or hours of video. In 5 years the ceiling rose roughly a thousandfold.
That growth changed what you build. Whole categories of prompt-juggling engineering from 2023 (conversation summarizers, sliding windows, elaborate chunk stitching) quietly became unnecessary for medium-sized problems. You just.. paste the thing in now.
Does a big window mean the model uses it all well?
No, and this is the practical catch. Research dubbed the failure “lost in the middle” (Liu et al., 2023): models recall material at the start and end of a long context better than material buried in the center. Vendors have improved on those findings, and needle-in-a-haystack retrieval scores now look great in launch posts, but real-world reasoning across a stuffed window is still weaker than over a short, relevant one.
There’s also attention dilution: pile in 400 pages of loosely related material and the model’s answers get vaguer, not sharper. A model given exactly the right 3 pages beats the same model given the filing cabinet.
Big context or RAG?
The wrong framing is “long context killed RAG“, which gets republished every time a window grows. The real trade:
- Cost: you pay per token, every call. Stuffing a million tokens into each request is technically possible and financially silly for repeated queries. Retrieval sends only what’s relevant.
- Latency: huge prompts take longer to process. Caching helps (providers now discount repeated prefixes), but retrieval is still leaner.
- Corpus size: most real document sets don’t fit in any window anyway. A million tokens is one big codebase, and roughly nothing next to a company wiki.
- Governance: retrieval can filter by user permissions and produce citations. A giant undifferentiated prompt can’t.
The working answer: small, static corpus that fits comfortably, just use context. Large, changing, or permissioned corpus, retrieve. Plenty of systems do both, retrieving generously because the window is roomy.
What are the benefits and drawbacks of big context windows?
Benefits of a large context window
Whole problems fit at once
A full codebase, a complete contract, a year of meeting notes: the model reasons over the entire thing instead of stitched fragments. Answers that need global view (contradictions, cross-references) get dramatically better.
Engineering you get to delete
Summarization chains, sliding windows, and chunk-stitching pipelines built for small windows become unnecessary code. Fewer moving parts, fewer places to be wrong.
In-context learning gets room
Dozens of worked examples in the prompt often rival a fine-tune, with none of the training infrastructure. The window is the cheapest customization surface there is.
Long conversations stop amnesia
Agents and assistants keep hours of working state in view. The “as I said earlier” failures fade with the scrolling.
Drawbacks of a large context window
You pay for capacity you fill
Per-token pricing means a stuffed million-token prompt costs real money on every call. Habitual over-stuffing is the new cloud waste.
Attention isn’t uniform
Middle-buried facts recall worse than start and end, and reasoning quality dips as irrelevant material piles up. Capacity is advertised; effective use is your problem to verify.
Latency grows with the prompt
Processing a giant context takes seconds before the first token appears. Interactive products feel it immediately.
It tempts you to skip curation
“Just paste everything” replaces thinking about what’s relevant, and quality quietly erodes. The old constraint was accidentally a discipline.
What should you remember when buying or building?
Treat the advertised window as capacity, and effective use of it as a separate claim to test on your own material. Watch the price per call at the sizes you’ll actually send, since “supports 1M tokens” and “affordable at 1M tokens” are different sentences. And design as if the window keeps growing, because it has every year so far; hard-coding today’s limit into your architecture is how you write next year’s migration ticket.