KV Cache
The KV cache is the working memory a model fills while reading your conversation — it grows with every token, and at long contexts it can rival the model itself in size.
While generating, a model keeps a record for every token it has seen so far — the keys and values of its attention layers, hence the name. That record is what lets token ten-thousand relate back to token three without re-reading everything.
The cost: the cache grows with context length, layer by layer. Your machine's memory has to hold the model's weights AND this cache at the same time, which is why a model that fits comfortably for short chats can fail on a long document.
Modern architectures attack exactly this. Some cache the full context on only a fraction of their layers, some use short sliding windows on the rest, and latent-attention designs compress each token's record into a small vector. Two models with the same parameter count can differ enormously in cache appetite — which is why reading it from each model's own published configuration beats any rule of thumb.
Where you see it on the radar
Our fit engine computes the KV cache layer by layer from each model's own config file — it is the middle number in every memory estimate, next to weights and overhead.
See the cache arithmetic per model →Related terms