A model with 70 billion parameters stored in standard 16-bit precision needs about 140 GB of memory just for its weights — more than most computers have. Quantization stores those weights with fewer bits, shrinking the model several times over so it fits on a single GPU, a laptop or even a phone. It is the main reason local AI is practical.

The basic idea

Neural network weights are numbers. During training they are usually stored as 16- or 32-bit floating-point values, which can represent a huge range very precisely. Most of that precision isn’t needed to run the model.

Quantization maps weights to a smaller set of values:

  • 8-bit halves memory compared with 16-bit, usually with negligible quality loss.
  • 4-bit cuts memory to about a quarter, with small but measurable loss for many models.
  • Lower (3-bit, 2-bit) saves more but degrades quality noticeably.

Rough rule: memory for weights ≈ parameters × bits ÷ 8. A 7-billion-parameter model needs about 14 GB at 16 bits, about 7 GB at 8 bits and about 3.5–4 GB at 4 bits (plus overhead).

How it preserves quality

Naively rounding every weight would hurt the model. Practical methods are smarter:

  • Group-wise scaling: weights are quantized in small groups, each with its own scale, so local ranges are captured precisely.
  • Error-aware methods: GPTQ (2022) quantizes weights layer by layer while compensating for the errors introduced, using a small calibration dataset.
  • Activation-aware methods: AWQ (2023) observed that a small fraction of weights matter much more, based on the activations they process, and protects those weights.
  • Mixed precision: keeping sensitive layers at higher precision.

Formats you will meet

  • GGUF: the file format used by llama.cpp and tools built on it, such as Ollama and LM Studio. Names like Q4_K_M or Q8_0 indicate the quantization type; “Q4” means about 4 bits per weight.
  • GPTQ and AWQ: popular for GPU inference with libraries such as vLLM and Transformers.
  • bitsandbytes: a library that quantizes models on the fly, used for example in QLoRA fine-tuning.
  • FP8 and FP4: low-precision floating-point formats supported natively by recent data-center GPUs for fast, efficient inference.

The trade-offs

  • Quality: the loss depends on model and task. Larger models often tolerate quantization better than small ones. Math, coding and long reasoning can be more sensitive than casual chat.
  • Speed: smaller weights move through memory faster, which often makes inference faster — memory bandwidth is usually the bottleneck when generating text.
  • Context memory: quantizing weights doesn’t shrink the memory used by long conversations (the KV cache), though some tools can quantize that too.

Choosing a level

For local use, 4-bit variants of the “K_M” type in GGUF are a popular balance of size and quality. If you have memory to spare, 8-bit is nearly indistinguishable from the original. Always test on your own tasks.

Common mistakes to avoid

  • Comparing models at different quantization levels. A 4-bit model versus an 8-bit model isn’t a fair comparison of the underlying models.
  • Forgetting context memory. Long prompts need extra memory beyond the weights; leave headroom.
  • Going below 4 bits by default. Very low-bit models can produce noticeably worse answers; use them only when memory forces you to.
  • Using untrusted quantized files. Download from reputable sources and prefer safe formats.

Quick guidance

Start with 4-bit or 8-bit versions for local use, and test quality on your own tasks before going lower.

Key takeaways

  • Quantization stores model weights with fewer bits to cut memory and often increase speed.
  • Memory for weights is roughly parameters × bits ÷ 8.
  • 8-bit is near-lossless for most models; 4-bit offers a good balance; lower levels degrade quality.
  • Methods such as GPTQ and AWQ preserve quality by compensating for quantization errors.
  • GGUF is the common format for local tools such as llama.cpp, Ollama and LM Studio.

Frequently asked questions

Does quantization make a model dumber?

Slightly, depending on the level. At 8 bits the difference is usually negligible; at 4 bits it is small for most tasks; below that it becomes noticeable.

What does Q4_K_M mean?

It is a GGUF quantization type: about 4 bits per weight (“Q4”) using llama.cpp’s “K-quant” scheme in a medium size-quality variant (“M”).

Can I quantize a model myself?

Yes. Tools in llama.cpp and other libraries convert and quantize models, but pre-quantized versions of popular models are widely available.