How Much VRAM Do You Need to Run LLMs Locally?
The central question when deploying a local LLM is straightforward: will it fit within your GPU's memory? The answer hinges on three key factors: model size, quantization level, and context length. This guide provides a practical starting point for determining the appropriate amount of VRAM for your needs.
How quantization affects VRAM
Quantization lowers the precision used to store model weights. Applying lower-bit quantization results in a smaller model footprint and reduced VRAM consumption, albeit with a slight trade-off in quality.
| Quant | Bits per weight | Typical use |
|---|---|---|
| Q8_0 | 8 | Very high quality |
| Q6_K | ~6.6 | Very good quality |
| Q5_K_M | ~5.5 | Good quality and size |
| Q4_K_M | ~4.5 | Good balance of size and quality |
| Q3_K_M | ~3.5 | Lower VRAM, more quality loss |
Q4_K_M is a popular choice when VRAM resources are constrained. If you have additional VRAM available, opting for Q5 or Q6 allows you to run the same model with less aggressive quantization.
Approximate VRAM by model size
The figures below are rough estimates for model weights. The actual VRAM requirement is higher because the runtime, KV cache, and context window also consume memory.
| Model size | Q8_0 | Q6_K | Q4_K_M | Q3_K_M |
|---|---|---|---|---|
| 4B | ~5 GB | ~4 GB | ~3 GB | ~2.5 GB |
| 8B | ~9 GB | ~7 GB | ~5.5 GB | ~4.5 GB |
| 12B | ~13 GB | ~10 GB | ~8 GB | ~6.5 GB |
| 14B | ~16 GB | ~12 GB | ~9 GB | ~7.5 GB |
| 27B | ~30 GB | ~22 GB | ~17 GB | ~13 GB |
| 32B | ~36 GB | ~27 GB | ~20 GB | ~16 GB |
| 70B | ~80 GB | ~60 GB | ~42 GB | ~34 GB |
These are estimates rather than hard limits. Different model architectures and quantization formats can influence the actual memory size.
What different amounts of VRAM can run
| VRAM | Practical range | Current examples |
|---|---|---|
| 8 GB | Small models around 4B to 9B | Gemma 4 E4B, Qwen3.5 9B |
| 12 GB | Small to mid-sized models around 9B to 14B | Gemma 4 12B, Qwen3.5 9B |
| 16 GB | 12B to 27B with lower quantization | Gemma 4 26B-A4B, Qwen3.6 27B at Q4 |
| 24 GB | 27B to 35B at Q4 to Q6 | Qwen3.8 27B, Gemma 4 31B |
| 32 GB | 27B to 35B at higher quantization | Qwen3.8 27B, Gemma 4 31B |
| 48 GB | Large dense models at lower quantization | 70B-class models at Q3 to Q4 |
| 80 GB | Large dense models at higher quantization | 70B-class models at Q4 to Q6 |
These ranges apply to models whose weights can fit on the GPU. Large MoE models behave differently: while only some of their parameters are active for each token, the model must still store its full set of weights. Consequently, a model with 100B or more total parameters will not fit in a 100B-sized VRAM budget simply due to having fewer active parameters.
MoE models
Mixture-of-Experts (MoE) models contain multiple groups of parameters known as experts. Since only a subset of these experts is activated per token, inference can be more efficient compared to a dense model with an equivalent total parameter count.
However, the inactive experts remain part of the model structure. Therefore, large MoE models may require significantly more memory than their active parameter count implies. Very large models may necessitate multiple GPUs or offloading to system RAM.
Context length also uses VRAM
Model weights represent only a portion of the total memory requirement. The KV cache expands as the context length increases; thus, running the same model at 64K context may demand substantially more VRAM than at 4K.
- Longer contexts require more VRAM.
- KV-cache precision impacts memory usage.
- Batch size and the number of concurrent users also increase memory consumption.
- Reserve some VRAM for the runtime rather than filling the GPU entirely with model weights.
Practical tips
- Verify the actual size of the quantized model you intend to run.
- Avoid treating the model file size as the exact VRAM requirement. Allow space for the KV cache and runtime.
- If a model does not fit entirely in VRAM, part of it can be offloaded to system RAM, though this typically results in slower inference.
- For long-context or agentic workloads, allocate more VRAM than the model weights alone require.
- Multiple GPUs can be used to split a model when a single GPU lacks sufficient VRAM.
Run it on DaDesktop
You do not need to purchase a GPU to run a local LLM. DaDesktop provides a cloud desktop equipped with the necessary VRAM, enabling you to run models directly without owning the hardware.
Select the VRAM tier that suits your model, load it, and begin usage. There is no setup, no hardware purchase, and no driver issues. See available GPUs for your options.