KV cache

Memory that stores the attention keys and values of tokens already processed, so a model does not recompute them when generating each new token.

Generating text token by token would be slow if the model reprocessed the whole conversation every time. The KV cache keeps intermediate results instead, trading memory for speed.

For long contexts the cache can take more GPU memory than the model weights, which is why serving systems manage it carefully, for example with paged memory.