Quantization
Quantization stores a model's weights at lower precision — for example 4-bit instead of 16-bit — cutting memory to roughly a quarter at a small quality cost.
A model is billions of numbers. Stored at full training precision, each number takes two bytes; quantization rounds them to a coarser scale so each takes only a fraction of that. Think of rounding every price in a supermarket to whole euros: the totals stay almost right, and the notebook you need becomes much smaller.
This is the technology that makes local AI possible at all. A 30-billion-parameter model needs on the order of 60 GB at full precision — beyond any consumer graphics card. The common 4-bit quantizations bring it to roughly a third of that, into the range real machines have.
The cryptic names — Q4_K_M, Q8_0, IQ4_XS — are recipes: how many bits, and which packing method. Higher numbers keep more precision and more gigabytes. For everyday chat use the popular 4-bit recipes hold up well; very aggressive 2- and 3-bit versions save more memory but the rough edges start to show.
Where you see it on the radar
Our fit tables measure the Q4_K_M file where one exists — the quantization most local-AI guides assume — and every table names the exact recipe it measured.
Check a quantized model against your hardware →Related terms