“Will this model fit on my GPU?” is the first question in any self-hosting project. The answer depends on more than the model’s size. This guide gives you rules of thumb to estimate memory for inference and fine-tuning, so you can choose hardware before spending money.

Component 1 — Model weights

The biggest fixed cost. Estimate:

weights memory ≈ number of parameters × bytes per parameter

Precision Bytes per parameter 8B model 70B model
16-bit (FP16/BF16) 2 ~16 GB ~140 GB
8-bit 1 ~8 GB ~70 GB
4-bit ~0.5 ~4–5 GB ~35–40 GB

See our guide to quantization for what these formats mean.

Component 2 — The KV cache

During generation, the model stores keys and values for every token in the context — the KV cache. Its size grows with:

  • context length (tokens in prompt plus output);
  • number of simultaneous requests;
  • the model’s number of layers and attention heads.

For a single chat with a moderate context, it may need a few gigabytes. For a server handling many long conversations at once, the KV cache can require as much memory as the weights — or more. Modern architectures reduce it with techniques such as grouped-query attention, and serving engines manage it efficiently, as explained in our inference serving guide.

Component 3 — Overhead

Runtime buffers, activations and fragmentation add more. Reserve 10–20% extra as a safety margin.

Worked example: running an 8B model

  • 16-bit weights: ~16 GB
  • KV cache for one user with a few thousand tokens: ~1 GB (varies by architecture)
  • Overhead: ~2 GB

Total ≈ 19 GB: fits on a 24 GB GPU. At 4-bit, the same model fits comfortably on an 8–12 GB card or a laptop.

Worked example: running a 70B model

  • 16-bit weights: ~140 GB — requires multiple 80 GB data-center GPUs.
  • 4-bit weights: ~35–40 GB — fits on a single 48 GB GPU with limited context, or two 24 GB GPUs.

Fine-tuning needs much more

Training stores gradients and optimizer state in addition to weights. With the Adam optimizer, full fine-tuning in mixed precision needs on the order of 16 bytes per parameter or more — over 100 GB for an 8B model. That is why parameter-efficient methods matter:

  • LoRA trains small adapters, dramatically reducing gradient and optimizer memory.
  • QLoRA loads the base model in 4-bit, making it possible to fine-tune large models on a single GPU.

See our LoRA fine-tuning overview.

Practical tips

  1. Check the model card for parameter count, architecture and recommended hardware.
  2. Decide context length and concurrency first; they drive the KV cache.
  3. Prefer quantized models for local and cost-sensitive use.
  4. Test before buying: rent a cloud GPU for an hour to measure real memory use.

Common mistakes to avoid

  • Counting only the weights. Context length and concurrency can double memory needs.
  • Assuming fine-tuning fits where inference fits. Training needs several times more memory unless you use LoRA or QLoRA.
  • Maxing out memory. Running at 100% leaves no room for longer prompts and causes out-of-memory errors.
  • Ignoring bandwidth. Two GPUs connected slowly can be much slower than one large GPU.

A quick sizing example

A 13-billion-parameter model in 16-bit precision needs about 26 GB for weights; at 4-bit quantization, roughly 7 GB, plus memory for the context cache.

Related: quantization explained.

Bottom line

Size for weights plus context, then add headroom.

Key takeaways

  • Weights memory ≈ parameters × bytes per parameter: 2 bytes at 16-bit, 1 at 8-bit, about 0.5 at 4-bit.
  • The KV cache grows with context length and concurrent users and can rival the weights in size.
  • Add 10–20% overhead as a safety margin.
  • Full fine-tuning needs roughly 16+ bytes per parameter; LoRA and QLoRA reduce this drastically.
  • Rent a GPU briefly to measure real usage before buying hardware.

Frequently asked questions

Can I run a 70B model on a consumer GPU?

Only with heavy quantization and usually across multiple GPUs or with parts offloaded to system memory, which is slower. Smaller models are more practical on consumer hardware.

Does Apple Silicon count as GPU memory?

Apple Silicon uses unified memory shared by CPU and GPU, so a Mac with large unified memory can load surprisingly large quantized models.

Why do models list different context lengths?

Longer contexts require more KV cache memory and compute; providers choose limits based on training and serving costs.