Local LLM VRAM Requirements: The Math Behind Which Models You Can Run

Search for which local model to run and you'll find a dozen ranked lists, all of which were accurate on the day they were published and are now quietly wrong. Models ship monthly. The arithmetic that determines whether one fits on your hardware has not changed in years, and once you can do it in your head you never need the lists again.

This is that arithmetic. It's three numbers added together, and the third one is the one everybody forgets.

The formula

Memory needed to run a model is:

total = weights + KV cache + overhead

Weights is the big term and the easy one:

weights (GB) = parameters (billions) × bytes per parameter

Bytes per parameter is set entirely by quantisation:

Q4 quantisation is why local inference is possible at all. It cuts memory by roughly 3.5× against FP16 for a quality loss that is real but small — and the alternative on consumer hardware isn't a better model, it's no model.

Note that Q4_K_M averages closer to 4.8 bits than to 4. K-quants keep some tensors at higher precision because degrading them hurts disproportionately, so the effective rate is always above the nominal one. That's why 0.6 GB per billion parameters is the rule of thumb worth memorising rather than the tidier 0.5. It checks out against real files: Q4_K_M puts a 7B model at ~4.1 GB, a 13B at ~7.9 GB, and a 70B at ~40 GB — 0.57 to 0.61 GB per billion in every case.

The number people forget: KV cache

Weights are fixed. The KV cache is not, and it's what turns "it loaded fine" into an out-of-memory error twenty minutes into a long conversation.

As a model generates, it stores a key and value vector for every token in the context so it doesn't recompute them. That storage grows linearly with context length:

KV cache (GB) ≈ 2 × layers × kv_heads × head_dim × context_length × bytes_per_element / 1e9

You rarely need to compute it precisely. What you need is the shape of it: KV cache scales with context length, and it can rival the weights themselves at long context. A 9B model with a 4096-token window might use under a gigabyte of cache. The same model at 128k context can need more memory for cache than for weights.

This is the practical consequence: raising OLLAMA_CONTEXT_LENGTH from 4096 to 32768 is not free. It's frequently the reason a model that ran yesterday won't run today.

Overhead — the CUDA context, activations, framework allocations — is typically 0.5–1.5 GB. Round up.

Worked examples

A machine with 8 GB of VRAM, running a 9B model at Q4 with default context:

weights   9 × 0.60  = 5.4 GB
KV cache  4k context ≈ 0.5 GB
overhead              ≈ 1.0 GB
                      -------
total                 ≈ 6.9 GB   ✓ fits, without much room

The same card attempting a 20B model at Q4:

weights  20 × 0.60  = 12.0 GB
                      -------
total                 > 13 GB    ✗ doesn't fit

A 24 GB card running a 27B model at Q4 with a generous 16k context:

weights  27 × 0.60  = 16.2 GB
KV cache  16k       ≈  2.0 GB
overhead            ≈  1.0 GB
                      -------
total               ≈ 19.2 GB    ✓ fits

That last one is the sweet spot of consumer local inference right now, and the reason 24 GB cards are the enthusiast default.

A shortcut when you don't want to do any of this: the model's download size is a good proxy for its weight memory. Add a couple of gigabytes and you have your answer.

What happens when it doesn't fit

Different tools fail differently, and the difference matters.

Ollama splits the model — as many layers as fit go on the GPU, the rest run on CPU. It works, and it is dramatically slower, because every token now crosses the PCIe bus. ollama ps tells you: the PROCESSOR column shows 100% GPU, 100% CPU, or a split like 40%/60% CPU/GPU. A split isn't an error, but if you see one and performance disappoints, that's your explanation.

Production servers like vLLM don't do this. They expect weights and cache to fit in VRAM and fail to start otherwise. This catches people migrating: a model that "worked" on Ollama via CPU spillover simply refuses to load. Worth knowing before you plan a move off Ollama.

Apple Silicon is the genuine exception to all of this. Unified memory means the GPU addresses system RAM directly, so a 32 GB MacBook runs models that would need a 24 GB discrete card on a PC. It's the best value in local inference and it isn't close.

Mixture-of-experts broke the old intuition

For years, parameter count predicted both memory and speed. Several current models — gpt-oss:20b, gemma4:26b, qwen3-coder:30b — no longer work that way.

A mixture-of-experts model holds all its parameters in memory but routes each token through only a small subset of them. A 26B MoE model might activate three to four billion parameters per token. The consequence:

• Memory scales with total parameters. You need the RAM for all 26B.
• Speed scales with activated parameters. It generates at roughly 7B-class speed.

So these models need a big card and run surprisingly fast on it. If you've been avoiding a 26B model because you assumed it would crawl, check whether it's MoE first — the assumption may be a generation out of date.

Choosing quantisation

Given a fixed amount of memory, you're choosing between a bigger model quantised harder and a smaller model quantised lightly. The general finding is that a larger model at Q4 usually beats a smaller model at Q8 for the same memory budget.

Practical guidance:

• Q4_K_M — the default, and correct for almost everyone. Best quality-per-gigabyte.
• Q5 / Q6 — worth it if you have headroom and are doing quality-sensitive work.
• Q8 — near-lossless, rarely worth the memory over Q4 unless you're evaluating quantisation damage itself.
• Q3 and below — degradation becomes noticeable, particularly on reasoning and code. Use only to squeeze a model onto hardware that genuinely can't hold Q4.

Sizing table

Using ~0.6 GB per billion parameters at Q4, plus context and overhead:

Model names will change. The left column won't.

Frequently Asked Questions

How much VRAM do I need to run a 7B model?

About 5 GB for the weights at Q4 quantisation, plus roughly 1–1.5 GB for context and overhead — so 6–7 GB total, which fits an 8 GB card comfortably. At FP16 the same model needs about 14 GB, which is why quantisation is effectively mandatory on consumer hardware.

Does quantisation make a model worse?

Yes, measurably, but less than people expect at Q4 and above. Q4_K_M typically costs a small amount of accuracy on reasoning and code while remaining clearly useful. Below Q3 the degradation becomes obvious. The comparison that matters isn't Q4 against FP16 — it's Q4 against not running the model at all.

Why is my model running slowly even though it loaded?

Almost always because it didn't fully fit in VRAM and part of it is executing on CPU. Check ollama ps — the PROCESSOR column shows the split. Every layer on the CPU means data crossing the PCIe bus per token, which dominates generation time. Move to a smaller model or a harder quantisation so the whole thing fits.

Does context length affect VRAM usage?

Significantly. The KV cache grows linearly with context length, and at long context it can exceed the memory used by the weights themselves. Raising a context window from 4k to 32k tokens can add several gigabytes. If a model loads but crashes partway through a long conversation, this is nearly always why.

Can I split a model across two GPUs?

Yes. Ollama distributes layers across multiple GPUs automatically, and production servers support tensor parallelism, which splits individual layers across cards for better efficiency — see tensor parallelism on Kubernetes. Two 24 GB cards give you roughly 48 GB of usable capacity, though interconnect bandwidth becomes a factor for the largest models.

See also

• What Is Ollama? — the runtime most people use for local models.
• Run an LLM Locally with Ollama — hands-on install and first model.
• When to Move Off Ollama — where local inference stops scaling.
• Running Llama 3 70B on Kubernetes — AWQ quantisation and tensor parallelism at production scale.

For shrinking what you send the model rather than the model itself, see JSON compression for LLM token optimisation.

Sizing hardware for an inference workload and want the numbers checked before you buy? Talk to us at Coding Protocols — we plan GPU capacity against real context lengths and concurrency, not vendor benchmarks.

Official References

• vLLM documentation — serving, batching and parallelism options
• Scheduling GPUs — device plugins and GPU resource requests