AI Glossary
Storing a model's numbers with fewer bits, for example 4 or 8 instead of 16, so it uses less memory and runs faster with a small loss in quality.
A 70-billion-parameter model needs about 140 GB at 16-bit precision but roughly 35–40 GB at 4-bit, putting it within reach of high-end workstations. Formats such as GGUF and methods such as GPTQ and AWQ are common for local models.
Quality drops become noticeable at very low bit widths, so test on your own tasks. See quantization explained.
Quantization shrinks AI models by storing weights with fewer bits. How 8-bit and 4-bit formats work, GGUF, GPTQ and AWQ, and the quality trade-offs to expect.
thelatentdaily.com Editorial Team3 min read