Context Windows and KV Caching Explained: Why They Drive AI Inference Costs

Context windows raise cost because attention compares each new token against every prior token, scaling quadratically rather than linearly. KV caching stores previously computed key and value vectors so they need not be recalculated, trading memory for speed, but that cache grows with sequence length, driving up GPU memory costs and limiting affordable context size.

1. What a token actually is

Before any of this makes sense, it helps to be precise about what a token is, since the whole article rests on it.

A model does not read text the way a person does, character by character or word by word. Text is first broken up into tokens, which are chunks of text a model treats as its basic unit of processing. A token is often a whole common word, sometimes a fragment of a longer or less common word, and sometimes a single character or punctuation mark. The word “caching” might be one token or split into two, depending on how frequently that exact sequence of characters shows up in the data the tokeniser was built from. This is why word count and token count are never quite the same number, and why unusual words, made up terms, or non English text tend to use up more tokens than plain English of the same length would suggest.

Every single thing described later in this piece, the context window, the quadratic cost of attention, the KV cache, is defined and measured in tokens rather than in words or characters. So when a provider advertises a context window of one hundred thousand tokens, or charges a price per token, they are talking about this exact unit. Getting comfortable with tokens as the base currency of everything that follows makes the rest of the piece far more concrete.

2. The core problem

Every time a language model generates a token, it has to look back over everything that came before it. That is not a simplification for beginners. It is the literal mechanism. Each new token is produced by comparing it against previous tokens in the sequence, deciding how much attention to pay to each one, and blending that information together.

This sounds fine until you separate two things that get lumped together in most explanations. Processing a fresh prompt for the first time is a different problem from generating each subsequent token once that prompt has already been processed. Those two stages have very different cost profiles, and understanding the difference is the whole point of this piece. But before getting into cost, it is worth being precise about what a context window actually is and why it exists at all, because most explanations skip straight past this.

3. What a context window is, why it is needed, and where it fits

A context window is the fixed number of tokens a model can hold and attend across in a single request. It is the entire space available for everything involved in producing the next token: the system prompt, the conversation history, any documents or retrieved content you have supplied, tool outputs, and everything the model has already generated in this response so far. All of that shares one budget.

The reason this exists at all comes down to a fact about how these models work that is easy to forget once you are using them through a chat interface. A language model has no persistent memory between requests. It does not remember your previous conversation because it stored it somewhere. It only knows about it because the entire conversation, or some representation of it, was resent to the model as part of the input for this request. The context window is the mechanism through which the model receives everything it is allowed to know about, for this one pass. Nothing outside that window exists to the model, no matter how relevant it might be.

This is not purely a design choice either. Attention, the mechanism that lets the model relate tokens to each other, is only trained and validated up to a certain sequence length. Positional encoding schemes, which tell the model where in the sequence a given token sits, are built around a maximum length the model was trained to handle. Push far beyond that and the model is not just slower, it is operating outside the range it learned to be reliable in. So the context window is both a practical constraint on how much you can send in, and a hard architectural ceiling on how far the model’s understanding of position and order can be trusted to hold up.

In terms of where it sits in the pipeline, the context window comes after tokenisation, which converts your raw text into the tokens the model actually operates on, and before the attention computation described in the rest of this piece. Every token that survives tokenisation, from every source you have fed into the request, has to fit inside this one shared space before the model can do anything with it. This is why a long system prompt, a large retrieved document in a RAG pipeline, or an extended tool call history can quietly eat into the room available for the actual conversation, even though none of those things feel like “the conversation” from a user’s point of view. They are all competing for the same fixed budget.

4. Prefill and decode

Inference happens in two distinct stages, and naming them properly clears up most of the confusion around context cost.

Prefill is the initial pass where the model processes the entire prompt for the first time and builds up its internal representations of every token in it. This is where a large prompt becomes expensive, because the model has to compute relationships across the whole thing before it generates a single word of output.

Decode is what happens after that. The model generates one token at a time, and for each new token it needs to attend back across everything already processed. The mechanism that makes this affordable, called KV caching, is covered next, since it changes the economics of decode substantially. Serving frameworks like vLLM and NVIDIA’s inference documentation treat prefill and decode as separate phases precisely because they behave so differently under load.

5. Enter KV caching

KV caching is the mechanism that prevents decode from repeating the quadratic work of prefill on every single step. KV stands for key and value, the two internal representations described below, and the technique is named directly after them.

When a transformer processes a token, it produces a query, a key, and a value for that token. The key and value are what every later token references when deciding how much attention to pay to this one. Without caching, generating each new token would mean recomputing the key and value for every earlier token in the sequence, every single time. That is enormous duplicated effort, and it is precisely the effort that made naive long context generation impractical before caching became standard.

KV caching stores the key and value for every token the first time they are computed, during prefill, and then keeps adding to that store as new tokens are generated during decode. Each new token only needs its own key and value calculated. Everything before it is already sitting in the cache, ready to be read rather than recomputed. This is what makes real time chat and long running agent loops practically possible.

6. The quadratic cost problem, and where it actually applies

This is the part that explains almost everything else in this piece, so it is worth getting precisely right.

Self attention computes a relationship between every pair of tokens in a sequence. For a fresh prompt of length n being processed for the first time during prefill, that means roughly n squared pairwise comparisons. Increase a fresh prompt from ten thousand tokens to one hundred thousand tokens and you are not looking at ten times the attention work during that initial processing. You are looking at something closer to one hundred times the attention work, because the calculation scales quadratically with prompt length.

That quadratic cost belongs to prefill. It is not the ongoing cost of every token generated afterwards. Once the prompt has been processed and its KV cache built, each new generated token during decode attends across the existing cache rather than recomputing the whole sequence from scratch. The cost of each individual decode step grows roughly linearly with how much context is currently active, not quadratically. The distinction matters because conflating the two makes long conversations sound catastrophically expensive in a way that is not quite accurate once caching is in play.

What is true, and this is the more interesting economic story, is that a large active context still makes every decode step more expensive than a short one, still consumes a growing amount of GPU memory, and still reduces how many concurrent requests a provider can serve on the same hardware. Long context has a real and compounding cost. It just is not the same shape of cost as the initial quadratic blow up during prefill.

7. Memory and bandwidth, not just memory

KV caching does not eliminate the cost of long context. It trades a large amount of repeated computation for memory consumption, and that trade has consequences worth understanding properly.

It is worth being precise here rather than treating this purely as a memory story. The cache still has to be read for every new token generated, and as the cache grows, the model has to read and process more cached keys and values for each new query. That read cost is a major reason long context decoding becomes increasingly constrained by memory bandwidth, not only by memory capacity. The cache itself has to live somewhere, and for models running on GPUs, that somewhere is GPU memory, which is finite and expensive. Every additional token of active context adds another entry that has to sit in memory, and has to be read from that memory, for the life of the generation.

A long conversation, a long document, or a long agentic trace does not just cost more compute time during its initial processing. It occupies more memory for longer, it demands more memory bandwidth on every subsequent step, and that memory is a shared, constrained resource a provider is allocating across every concurrent request they are serving. NVIDIA’s own serving guidance treats KV cache capacity as one of the primary constraints on GPU throughput, and research on KV cache optimisation consistently identifies long context as a major bottleneck for exactly this reason.

The sentence this is really building towards is simple. Context consumes serving capacity even after you have eliminated most of the redundant computation. That is the FinOps insight underneath everything else in this piece.

8. Why this matters for model routing and cost

This is where the theory stops being academic and starts affecting decisions you actually make when choosing between models and providers.

A workload that generates short responses to short prompts behaves very differently, economically, from a workload that maintains long running active context across many turns. If you are routing between models through something like OpenRouter, or choosing between a smaller model like DeepSeek and a larger frontier model, the context length of your typical workload changes which option is actually cheaper, not just which one is technically capable of the task.

A smaller model carrying a large active context can, in some cases, cost more to serve than a larger model with a short one, because active context drives memory and bandwidth consumption independently of model size. This is a genuinely counterintuitive result for people who assume model size is the only lever that matters for cost. Context length is an independent lever, and for agentic workflows that accumulate long histories, it is often the more important one.

9. Practical implications for engineers

A few things follow directly from all of this, and they are worth acting on rather than just understanding abstractly.

Prompt caching, offered by providers as a way to mark a portion of your prompt as reusable across requests, is best understood as the serving layer equivalent of avoiding repeated prefill work on an identical prefix, often through reuse of previously computed model state. It is closely related to the KV caching mechanism described above, though providers may manage, persist, or reconstruct that underlying state differently, so it is worth treating prompt caching as an abstraction rather than assuming it is always literally the same in memory cache used within a single generation.

The pricing makes the underlying economics visible in a way that is worth looking at directly. Anthropic currently charges a five minute prompt cache write at one and a quarter times the normal input rate, while a cache read costs a tenth of the normal input rate, and states that the cache becomes worthwhile after just one read. OpenAI describes prompt caching as capable of reducing input token costs by up to ninety percent and latency by up to eighty percent for repeated prefixes. Google’s Gemini API offers a similar implicit caching mechanism for repeated prompt prefixes. Those are not marketing numbers dressed up to sound impressive. They are the infrastructure economics described in this piece showing up directly in a price list.

Sliding window attention, used by some model architectures, deliberately limits how far back a token can attend, trading some long range comprehension for a hard cap on how large the active context and its cache can grow. Knowing whether a model you are evaluating uses full attention or a windowed variant, or some hybrid of the two as several current serving frameworks now explicitly support, tells you something real about its behaviour on long documents, not just its marketing claims about context length.

And when you are designing an agentic system that accumulates a long working history of tool calls, retrieved documents, and reasoning steps, the context window is not a soft constraint you can ignore until you hit the wall. Prefill cost, decode cost, and cache size are all quietly increasing from the very first turn. Designing for context hygiene, trimming, summarising, or offloading history rather than letting it accumulate indefinitely, is not a nice to have. It is the difference between a system that scales and one that quietly becomes unaffordable.