How Much VRAM Do You Need for Local AI?
VRAM (video memory on your GPU) is the single biggest factor in what AI models you can run locally. Here's a simple way to estimate what you need.
The rule of thumb
A model's weights need roughly number of parameters × bytes per parameter:
- 16-bit (FP16/BF16): about 2 bytes per parameter — a 7-billion-parameter model needs roughly 14GB just for weights.
- 8-bit: about 1 byte per parameter — roughly 7GB for a 7B model.
- 4-bit quantised: about 0.5–0.6 bytes per parameter — roughly 4–5GB for a 7B model.
On top of the weights you need extra memory for the context window (the KV cache), which grows with longer prompts and conversations, plus some overhead for the software itself. Adding 20–30% headroom is a sensible starting point.
Approximate examples (4-bit quantised, inference)
| Model size | Weights (approx.) | Practical VRAM target |
|---|---|---|
| 7–8B parameters | ~4–5GB | 8GB or more |
| 13–14B parameters | ~7–8GB | 12–16GB |
| 30–34B parameters | ~17–20GB | 24GB or more |
| 70B parameters | ~35–40GB | 48GB or more (often across multiple GPUs) |
These are rough estimates for planning only. Actual requirements vary by model architecture, quantisation method, software and context length.
Training needs much more
Fine-tuning or training needs memory for gradients and optimiser states as well as weights — often several times more than running the same model. Techniques like LoRA reduce this considerably.
What if I don't have enough VRAM?
- Use a smaller or more heavily quantised model
- Reduce context length
- Split the model across multiple GPUs
- Offload some layers to system RAM (slower)
Not sure what you need? Ask us for a recommendation or browse GPUs and custom AI workstations.