AI Glossary
Running a trained model to produce an output, such as answering a prompt, as opposed to training it.
Every chatbot reply is an inference request. Its cost depends on model size, input and output length, and hardware efficiency; providers usually charge per million tokens. As AI usage grows, inference rather than training accounts for most of the computing spent on AI.
Techniques such as batching, caching and quantization make inference faster and cheaper. See how LLM inference serving works.
Quantization shrinks AI models by storing weights with fewer bits. How 8-bit and 4-bit formats work, GGUF, GPTQ and AWQ, and the quality trade-offs to expect.
thelatentdaily.com Editorial Team3 min read
What happens when a server answers thousands of LLM requests: prefill and decode, the KV cache, continuous batching, PagedAttention and the metrics that matter.
thelatentdaily.com Editorial Team3 min read
Estimate the GPU memory needed to run or fine-tune a language model: weights, KV cache, activations and overhead, with worked examples for 8B and 70B models.
thelatentdaily.com Editorial Team4 min read