What is a context window in AI?

A context window (or context length) is the maximum number of tokens an LLM can process in a single request — everything the model can 'see' while generating a response, including the response itself. It's the model's working memory for one request, not its long-term memory or its training data. The window is cleared after every request; chatbots only seem to remember conversations because they re-send the whole transcript each turn.

How many tokens fit in a context window?

It depends on the model and how your text tokenizes. As rough guides, 1 token is about 4 English characters or 0.75 words, so 100 tokens ≈ 75 words. 128K tokens ≈ 96,000 words (~300 pages of prose), and 1M tokens ≈ 750,000 words (~2,500 pages or about 8 novels). Code and non-English scripts tokenize less efficiently and count far higher per character.

What is the difference between context window and context length?

None, functionally. Context window and context length are near-synonyms: both refer to the token capacity a model can process in one forward pass. A related term, sequence length, is more granular — it refers to a single input or output sequence within that window, and both input and output count against the same window. Micron's explainer distinguishes them: window is the maximum capacity, length is how much you've actually put into it.

What consumes a context window besides my message?

Everything the model reads counts: the system prompt, tool and function definitions, the full conversation history (re-sent every turn), retrieved RAG chunks, attached files and images, and the space reserved for the model's reply. Product 'memory' still fits in the window when injected. Prompt caching reduces the cost of a repeated prefix but does not free window space.

How do I count tokens for my model?

Use the provider's official tooling against the exact model you'll call: OpenAI's tiktoken library or web tokenizer for GPT models, Anthropic's count_tokens API for Claude, and Google's countTokens API for Gemini. Tokenizer differences are real, so counts don't transfer between providers — the same text can be roughly 30% higher on some newer tokenizers.

Why do I get 'context window exceeded' on a short message?

Because the model reads the entire conversation, not just your last line. Long chat history, a bloated system prompt, tool definitions, and retrieved documents all count every turn. That's why long chats (or agent runs) creep toward the limit even when each individual message is short. Start a fresh conversation, trim old turns, or summarize the history to free space. A bigger model can help, but retrieval and compression are usually the better fix.

What is the largest context window in 2026?

Among widely available models, Gemini 1.5 Pro formally supports up to 2M tokens, while the current frontier list — Gemini 2.5 Pro, GPT-5.6 (Sol/Terra/Luna), Claude Opus 4.8 and Sonnet 5 — offers around 1M tokens. Open-source Meta Llama 4 Scout lists a 10M-token window on its model card, though hosts usually cap it lower and effective quality there is untested in production. These figures change with every release; confirm the current number on the official model page.

Should I use a bigger context window or RAG?

Use a long context when you need to reason across a single large artifact in one pass — one contract, one transcript, one codebase. Use RAG when your corpus is bigger than the window, changes over time, or must cite specific passages — retrieval keeps the prompt small, which is faster, cheaper, and often more accurate. Most production systems combine both, usually a stable cached system prompt plus retrieved chunks.

What is the 'lost in the middle' effect?

A well-replicated finding from Stanford's Liu et al. study: language models recall facts placed at the start and very end of a long prompt reliably, but perform worse on information in the middle — sometimes 30%+ worse. It's why a full context window isn't a healthy one, and why placing your most important instruction at the start of the system prompt matters.

What Is a Context Window? Tokens, Limits, and the 1M Context Truth

Q: Is a bigger context window better?

Not automatically. A larger window accepts more input, but accuracy and recall typically degrade as input grows — the 'lost in the middle' effect and 2025 context-rot research show models drop in quality long before the advertised limit. Bigger windows also cost and run slower. The headline number is a ceiling, and the usable length is often 2–10x smaller. Retrieve only what's relevant (RAG) instead of stuffing the window.

Every heavy user of AI chatbots eventually runs into the same wall. You paste a long document, or your chat has been going for an hour, and instead of an answer you get a red error: “This model’s maximum context length is … tokens,” or “prompt is too long.”

That error is the most visible symptom of the single most fundamental concept in language models: the context window. And it’s probably the most misunderstood number in AI. Everyone can recite “128K” or “1M tokens” without being able to say what actually lives inside that space, why it’s bounded, or why cramming it full usually makes answers worse.

So, in one sentence: A context window (or context length) is the maximum amount of text — measured in tokens — that an LLM can process in a single request: everything the model can “see” at once, including the response it’s about to write.

This is the explainer. We’ll cover what tokens actually are, what silently fills the window, the engineering limits behind it, how the major 2026 models compare, the gap between advertised and usable context (yes, that’s real), and a decision framework for sizing a window you actually need. Everything is grounded in official documentation, linked at the end.

What a context window actually is

A large language model has no memory of its own. Each time you send a request, the model reads a block of text, predicts a response, and forgets everything. The context window is the maximum size of that block — the total amount of text the model can consider at once for a single request, measured in tokens.

It’s the model’s working memory: a short-term desk it must place everything on for each task, then clear when the task ends. Unlike long-term memory, it resets every call.

Two things about this trip people up.

The window holds both what you send and what the model generates. Input tokens and output tokens share the same budget. If you fill it to the brim with prompt, there’s no room left for the answer.
The window is per-request, not persistent. When a chatbot “remembers” your conversation, it’s because the app re-sends the entire history to the model on every single turn. That’s exactly why long chats creep toward the limit even when each individual message is tiny.

Everything that counts inside the window

You don’t own just your message: everything the model reads to answer counts against the same window. This is where most people get burned — the text in front of you is a tiny slice of the actual request.

What goes in	Rough cost (English)	Notes
Your latest message	10–1,000+ tokens	The headline, but rarely the biggest line item
System prompt	100–2,000+ tokens	Sent on every request — bloat here is silent
Tool / function definitions	200–2,500+ tokens	Full JSON schemas for every tool, every turn
Conversation history	grows without bound	Re-sent in full every turn
Connected files / attached documents	1,000–15,000+ tokens	A pasted PDF can be the whole budget
Retrieved RAG chunks	1,000–12,000+	Only when your app injects retrieval
Extended “thinking” (reasoning)	up to the max output cap	Counts as output; thinking blocks stay in the window on some models
The model’s reply	up to max output	Reserved from the same budget

Two details stand out if you read the official docs carefully:

Prompt caching does not free space. Anthropic’s docs note cached prefixes still occupy the context window. Caching changes what you pay; it doesn’t change whether it fits.
Multimodal input is token-expensive. Google’s guidance: roughly 258 tokens per small image (≤384px), ~263 tokens per second of video, and ~32 tokens per second of audio. A video clip can swallow a window nearly alone.

That’s the “desk” that Anthropic’s diagram shows: as turns accumulate, each assistant response stays in the window until the whole conversation approaches the limit.

Tokens, the real unit of the window

Context windows aren’t measured in words or characters — they’re measured in tokens, the sub-word units a tokenizer produces. A token can be a whole word, part of a word (e.g., tokenization → token, ization), a single character, or even a space.

Rough guidance, as the vendors define it:

1 token ≈ 0.75 English words, or ≈ 4 characters.
100 tokens ≈ 75 words.
128K tokens ≈ ~300 pages of prose.
1M tokens ≈ 750,000 words ≈ 8 novels (Google’s comparison).

But tokens are not universal. Every model tokenizes differently, and different tokenizers have different vocabularies. The same sentence can be, say, 120 tokens on one model’s tokenizer and 150 on another. That’s why token counts never transfer 1:1 between OpenAI, Anthropic, and Google.

Two practical consequences:

Code and non-English text cost more tokens per character. Indentation, camelCase, brackets, and symbols each split into their own pieces. Code and scripts like Chinese or Japanese routinely tokenize 1.5–2x+ higher than plain English prose for the same amount of meaning.
Count against the model you’re actually calling. A prompt that fit in GPT-4o’s window can overflow Claude’s after a model upgrade if you rely on the wrong tokenizer’s estimate — a real 2026 gotcha that surprises even experienced developers.

Why are context windows limited?

The ceiling isn’t a marketing decision — it’s a physics problem. Language models run on transformers — the architecture that powers nearly every modern LLM — where the core operation is self-attention: every token calculates a relationship with every other token in the sequence.

That attention cost is quadratic in sequence length — commonly written O(n²). Double the input, and you quadruple the attention work. A 4K-token window means roughly 16 million pairwise comparisons; a 100K-token window means around 10 billion. That quadratic is why a genuinely large window costs so much to serve, and why providers price both input and output per token.

Two other costs follow:

Latency. Prefill (reading the prompt) takes time proportional to the input; generation (“decode”) is slower and sequential — each output token depends on everything before it. Long inputs add modest delay; long outputs add the real wall-clock time.
KV cache. During output the model holds a key/value (KV) memory of the entire context in high-bandwidth memory (HBM), and that cache grows with context length. Push it out of fast memory and throughput collapses — which is why a consumer GPU is far slower near the 128K ceiling than the marketing suggests.

That’s also why providers price long-context requests specially and why prompt caching (which lowers cost, not footprint) matters.

Context windows vs. max output tokens

A crucial, frequently-blurred distinction: the context window is the total budget (input + output). Max output tokens is a separate cap on what the model can generate in one turn.

Model group (2026)	Context window (total)	Max output
Claude Opus 5 / Opus 4.8 / Sonnet 5	1M tokens	128K
GPT-5.6 (Sol/Terra/Luna)	1.05M tokens	128K
Gemini 2.5 Pro	~1.05M tokens	65K
Claude Sonnet 4.5 / Haiku 4.5	200K (per Anthropic docs)	64K (Haiku)
Llama 3.1-70B (open)	128K	not standardized

This matters in two ways:

A model with 1M input but 8K max output can ingest a whole dataset but only write a short reply per turn.
Many API forms reserve output space: if you request max_tokens=128K on a 200K model, your usable input is only 72K. Long-context models explain this split.

So when you see a headline “1M context” — always check what the input cap is separately from output.

The gap between advertised and usable context

Here’s the part most explainers skip, and the reason I said a full window can hurt.

A context window number is the theoretical ceiling at which the model will reject input. But there are three well-documented ways models perform worse than the headline:

Lost in the middle. A Stanford/TACL-2024 study showed recall is strong for tokens at both ends of a long prompt, but dips ~30%+ for facts buried in the middle. Exactly where you’d place an important instruction — in the center of a huge stack — is where the model loses it.
Context rot (Chroma 2025). Chroma studied 18 frontier models and found every model declined in accuracy as tokens grew — even on easy tasks. Degradation isn’t a cliff near the limit; it’s a slope that starts far sooner.
Attention decay with distance. Positional awareness weakens as sequences lengthen, so the effective context — where the model reliably answers — is often a fraction of the advertised one.

The consequence is the opposite of instinct: a full context window is not a healthy window. If the answer lives in 8 relevant tokens, a request that buries them inside 200,000 noisy ones will often answer worse than the same task run on just those 8 tokens.

About the numbers. The Chroma (“context rot”) and Stanford “lost in the middle” results are their published findings, not our own measurements; the links are in Sources. What we add beyond that is a practical advisory — retrieve, don’t stuff — that follows directly from those findings.

Real choices: 128K vs 1M vs 10M

Providers now race window sizes upward — partly a real capability, partly a marketing number. The windows exist, but “the bigger the better” fails in practice. The winning move is picking the smallest window that fits the whole task — where “task” means one logical unit of work.

What do the headline sizes actually buy you?

Window	Realistic uses
8K–32K	Everyday chat, quick Q&A, RAG apps (retrieval keeps you small)
200K	Long documents, interview-transcript analysis, one book at a time
1M	Whole codebases, entire business contracts, transcripts of hours of calls
2M–10M	Extreme whole-repo / multi-doc analysis — only for corner cases

And the rule the industry keeps repeating:

One big document to reason over → long context.
A huge corpus of “let me ask anything” → RAG (retrieval), not 10M context.

Retrieval keeps your prompt tiny and your cost/latency low; stuffing 10M of code into one window is expensive and often less accurate than retrieving 4 relevant chunks.

What actually decides the “right” context window in 2026

Before you size anything, answer these three questions in order:

What is my task’s single-time working set? (Documents + history + tools + output needed together, at once?)
Is it one artifact or many? One → long window. Many and large → RAG. Both → hybrid.
What is my cost & latency budget? Input tokens are cheap, but the KV/attention cost of a 1M prompt is still real.

The output of that process is usually a far smaller window than 1M — because the highest-value tasks (reading a single 10-page contract, debugging a single failing file) fit in 200K comfortably.

Most production systems look the same: a large window is a safety valve, not an operating point. You want to operate far below it and leave the top third empty.

Practical context management in 2026

Context engineering has replaced “prompt with as many tokens as you can afford.” Four moves that matter in practice — each one detailed in our other guides but worth naming here:

Retrieve, don’t stuff. Pull the few relevant chunks instead of pasting the whole corpus. Debug retrieval quality before blaming context — our RAG deep-dive covers this.
Budget the window on paper. Count input (system, history, tools, retrieval) + reserved output. If it’s >80% of the model, pause.
Cache what repeats. Prompt caching (Anthropic, OpenAI, Google all ship it) cuts the cost of a repeated prefix — but not the footprint, which still counts toward the window.
Isolate your sub-agents. Split a big task across smaller contexts — each sub-agent with a fresh window returning only its conclusion, what context engineering calls “isolate.”

The short version of a recommendation: Claude (Anthropic) for long single-document tasks, OpenAI GPT-5.6 for cost-efficient reasoning inside a 1M window (Sol/Terra/Luna tiered input pricing of $5 / $2 / $0.20 per million tokens), Google Gemini for multimodal large-context ingestion, and local open models (Llama) when you own the hardware and don’t need extreme length. All current as of August 2026.

Prices and qualities right now: prices are documented on official vendor pages (OpenAI publishes per-model pricing on its Models page), and all figures are as of August 2026 — always re-check the current page before committing to a workload.

FAQ

What is a context window? The maximum number of tokens an LLM can process in one request — what it can “see” at once, including its own answer. The window is working memory: cleared after every request.

Is a bigger context window better? Not automatically. Lost-in-the-middle, context rot, latency and output caps all rise with context length, so “bigger” can trade accuracy and cost. Use the smallest window that fits the whole working task.

What fills a window besides my message? System prompt, tool definitions, full chat history, attached files, retrieved chunks, and the reserved output space for the reply. Prompt caching doesn’t free space, only lowers cost.

How do I estimate tokens? Roughly 1 token ≈ 0.75 words. Count against the exact model: OpenAI tiktoken, Anthropic count_tokens, Google countTokens. Counts don’t transfer between providers.

Which model should I pick for a large context? For a single massive artifact: Claude 1M or Gemini 2.5 Pro 1M. For long agent work: GPT-5.6 (cheap input at $0.20–$5/M). For self-hosted budgets: Llama open models. Check current official pages for the exact window and price.

Is context window = memory/knowledge? No. The context window is temporary working memory for a single request; long-term knowledge lives in the model’s weights and in product “memory” features (stored and re-injected). Confusing the two leads to over-pasting.

What’s the biggest mistake with context windows? Treating the advertised number as usable. Headline is a hard ceiling for accepting input; the usable length where quality holds is often far smaller. Design around usable, budget for the ceiling.

Keep reading

This is the foundation concept. The next logical stops:

Fix “context window exceeded” errors — the practical end of this.
Context engineering — how to curate what goes in.
How large language models actually work — from tokens to attention.
RAG, explained — the alternative to big windows.
Context vs. fine-tuning — when memory and retrieval beat a larger window.

Sources

Official documentation and primary research referenced in this article (all current as of August 2026):

Anthropic — Context windows · Effective context engineering for AI agents
OpenAI — Models (context windows & pricing) · How to count tokens
Google — Gemini API: long context · Tokens & counting
Meta — Llama 3 model card
Liu et al. — Lost in the Middle (Stanford, TACL 2024)
Chroma Research — Context Rot: how increasing input tokens impact LLM performance
IBM — What is a context window?
Stanford HAI — What is a context window?

The context window is where everything else in the AI era happens: it’s the workspace, the budget, and the battlefield at once. Once you stop thinking of it as a number “bigger is better” and start treating it as a working budget you design around — counting tokens, retrieving instead of stacking, keeping the middle relevant — the entire class of confusing errors starts making sense. That’s the single idea to take away.

Correction policy: figures and prices verified against official source pages in the Sources list; please flag any numbers that lagged a version when a newer model ships.