Context Windows Explained: AI Token Limits Matter

You paste a 50-page contract into ChatGPT and by the time it reaches the end, it has “forgotten” the beginning. That is not a bug: it is a context window limitation, and understanding it will save you money, mistakes, and a lot of frustration when you are choosing a model or building an AI system.

What a Context Window Actually Is

A context window is the amount of text a model can “see” at once: your input, its own output, and any instructions, all counted together. It is measured in tokens, not words. A token is roughly 0.75 words, so 1,000 tokens is about 750 words.

Current models range from around 8,000 tokens on the smallest, older systems to over 1 million tokens on the newest large language models. Once you exceed the window, the model either truncates your input, throws an error, or silently drops earlier content, none of which are outcomes you want in a production system.

Why It Matters for Your Business

This is not an academic detail. It decides whether a given tool can actually do the job you want it for.

• Document analysis. Can the model read your full 40-page contract, or only the first 15 pages before it starts guessing? Our AI contract analysis guide covers this in more detail for legal and procurement use cases.
• Conversation memory. In a chatbot or support agent, does it remember what the customer said five messages ago, or has that fallen out of the window?
• Retrieval-augmented generation. In a RAG system, the context window caps how many document chunks you can hand the model alongside a question. Bigger window, more chunks, better odds the answer is actually in there.

Get this wrong and you end up with a system that looks fine in a demo and fails quietly in production, once real documents and real conversations exceed what you tested.

Current Model Context Windows Compared

Numbers move fast in this market, but as of mid-2026 the practical picture looks roughly like this:

Two caveats matter more than the headline numbers. First, larger windows cost more per request, you are billed per token, so a 1M-token request is not free just because the model can technically handle it. Second, advertised limits and usable limits are different things. Recent benchmarking suggests effective accuracy typically holds up to only 60 to 70% of the advertised window before quality degrades meaningfully, per analysis from Coding Scape’s review of long-context models.

Why Bigger Isn’t Automatically Better

The temptation is to reach for the model with the biggest number and stop worrying about limits. Three reasons that is the wrong instinct:

Cost. Every token you send, including tokens you did not need to send, is billed. Stuffing 300K tokens of loosely relevant context into a request because you can is an expensive way to get a worse answer.

Speed. Longer inputs take longer to process. If your system needs a fast reply, a live chat widget, a booking assistant, sending unnecessary context slows down every single request.

Accuracy. This is the one people don’t expect. The landmark “Lost in the Middle” research by Liu et al. found that model accuracy follows a U-shaped curve across a long context: strong recall for information near the start and end, and a measured accuracy drop of over 30% for information buried in the middle. This pattern held across six different model families in testing. In practice, a model given a 40-document RAG result set is not treating all 40 documents equally, it is quietly favouring the first few and last few.

The practical implication: right-size the context you send, don’t maximise it. A well-curated 20K tokens of relevant material will usually outperform an undifferentiated 200K-token dump.

Working Within the Limit

When your data genuinely exceeds what a model can handle well, four patterns cover most business cases:

1. Chunking. Break large documents into smaller, logically bounded sections, by clause, by chapter, by section heading, rather than arbitrary character counts.
2. Summarisation chains. Summarise each chunk first, then feed the summaries (not the raw text) into a final synthesis step. This trades some detail for reliability at scale.
3. Selective retrieval (RAG). Instead of sending everything, retrieve only the chunks most relevant to the specific question being asked. This is the standard pattern for chatbots answering questions against a large knowledge base.
4. Hierarchical processing. For very large document sets, process at multiple levels, section summaries roll up into document summaries, which roll up into a final answer. Slower, but avoids the middle-of-context accuracy drop entirely.

A rough way to check whether you have a problem at all: take a typical input file, divide its word count by 0.75 to estimate tokens, and compare that against the model’s usable limit (roughly two-thirds of the advertised figure, per the caveat above). If you are close to or over that line regularly, you need one of the four patterns above, not just a bigger model.

Choosing the Right Setup for Your Business

None of this needs to be guesswork. If you are building or buying a system that touches contracts, long email threads, or a large knowledge base, context window behaviour should be part of the spec from day one, not something discovered after launch when a customer pastes in a document that breaks the whole flow. It is also directly linked to running costs; our breakdown of AI agent costs covers how token usage translates into a monthly bill.

Sources

• Lost in the Middle: How Language Models Use Long Contexts, Liu et al., arXiv, 2023 (accuracy degradation findings)
• LLMs with the Largest Context Windows, Coding Scape, 2026
• Long Context | Gemini API, Google AI for Developers, 2026
• AI Context Window Sizes 2026: Claude 200K vs GPT 128K vs Gemini 1M, Claude 5 Hub, 2026

Need AI architecture that handles large documents without losing information in the middle of the context? Get in touch with your use case, or read more about our AI systems engagements.