Context Window

What the context window holds

Language models keep no memory between requests; the context window is their entire working memory for one call. It is filled with tokens - roughly three-quarters of a word each in English - covering the system prompt, any prior turns of a conversation, documents supplied for retrieval, and the tokens the model generates in reply. When a task exceeds the window, the oldest tokens must be dropped or summarised, which is why long agent runs and large-codebase tasks are constrained by context size. Current open-weight models we serve, such as MiniMax M2.7, offer windows around 192K tokens.

During inference the whole input context is read in the prefill phase, where the model processes every prompt token in parallel to build its internal state - the KV cache - before it can emit the first token. Each token generated during decode then attends back over that stored context. So the context window is not just a capacity limit; it is the thing being processed on every request, and its size directly shapes how that request performs.

Why context length drives speed and cost

The cost of context is not linear. The attention mechanism at the heart of the transformer compares every token with every other token, so the work in prefill scales quadratically with prompt length: a 1,000-token prompt involves on the order of a million attention interactions, while 10,000 tokens involves a hundred million. In production terms, a 128K-token prompt can take several seconds to prefill, and a one-million-token prompt tens of seconds, before a single output token appears. This is why time to first token numbers are meaningless without the input length behind them.

Context also consumes memory, and memory is the binding constraint during generation. The KV cache holds the keys and values for every token in the window and grows with context length - for a large dense model it can reach tens of gigabytes at a 128K context. Reading that cache plus the model weights back for every generated token is a memory-bandwidth problem, not a compute one, which is why long-context generation is where GPU utilisation tends to collapse and where memory-centric hardware helps most. Methods such as PagedAttention manage the cache more efficiently to raise how many long-context requests a system can serve at once.

Reading context claims critically

A large advertised context window does not mean a model uses all of it well, or quickly. Two providers serving the same model with the same nominal window can differ sharply in how fast they prefill a long prompt and how gracefully they degrade under load - differences a headline number hides. When context matters to your workload, compare prefill time at your real input length and sustained output speed, not the maximum window alone. On our EU infrastructure, prefill runs on dataflow hardware designed to keep data on-chip, shortening the wait a long prompt imposes before the first token appears.

Related terms