How much VRAM do you actually need for local AI?

"Will this model run on my card?" is the question that comes up more than any other, and it has a genuinely simple answer that gets buried under a lot of noise. This article works through the arithmetic, the two numbers that matter, and the memory cost almost nobody accounts for until it bites them.

The short answer

Take the download size of the model file and add headroom for the conversation. That is your memory requirement, near enough. Everything below is an elaboration of that sentence.

Roughly, by tier:

• 8 GB — models up to about 9B at four-bit quantisation, and short conversations. Granite 4.2 8B is 5.3 GB. Workable, occasionally frustrating.

• 12 GB — the 12B class comfortably. Gemma 4 12B is 7.6 GB at Q4_K_M, leaving reasonable room for context.

• 16 GB — the practical entry point. gpt-oss-20b is 14 GB and OpenAI's own model card puts it inside 16 GB of memory.

• 24 GB — the sweet spot, and the tier most of the interesting 2026 models are designed around. Qwen3.8 27B is 18 GB at Q4_K_M; Meta's Muse Glimmer 30B is also 18 GB and was explicitly aimed at 24 GB and 32 GB consumer cards.

• 32 GB and up — the same models with much more context, or two of them loaded at once, or a step up in quantisation for better quality.

• Beyond 48 GB — unified-memory machines, multiple cards, or models that are more interesting than useful at home. The gpt-oss-120b model card asks for an 80 GB GPU.

Why memory and not compute

Generating text is memory-bound, not compute-bound. To produce each token the model reads through its entire set of weights. The GPU is rarely short of arithmetic; it is waiting on memory. That has two consequences worth internalising.

Capacity decides what runs at all. If the weights do not fit, the model either spills into system RAM — which works, but drops throughput off a cliff, because it is now reading across the PCIe bus — or fails to load.

Bandwidth decides how fast it runs. This is the number nobody quotes and it matters enormously. When Phoronix benchmarked llama.cpp on an RTX 5090 against an RTX 4090, text generation was about 1.58 times faster. The two cards' memory bandwidths are 1,792 GB/s and 1,008 GB/s — a ratio of 1.78. Prompt processing, which genuinely is compute-bound, improved by only around 17% across the same pair. That is about as clean a demonstration of the principle as you will find.

How quantisation changes the sum

Almost nobody runs models at full precision. Quantisation stores each weight with fewer bits, and the file size falls roughly in proportion. Unsloth publishes a memory ladder for Qwen3.8 27B that shows the shape of it clearly: around 16 to 19 GB at four-bit, 23 to 26 GB at six-bit, 31 GB at eight-bit, and 56 GB at BF16. Meta gives the same picture for Muse Glimmer 30B — over 55 GB at full precision, under 20 GB at roughly four bits.

Four-bit, specifically the Q4_K_M variant, is the default nearly everywhere and for good reason. The quality cost is real but small, and it is the difference between a 27B model running on a card you own and not running at all. Moving to eight-bit roughly doubles your memory bill for a difference most people cannot reliably detect in ordinary use.

The corollary is worth stating plainly: for a fixed memory budget, a bigger model at four-bit almost always beats a smaller model at eight-bit.

The bit everyone forgets: the KV cache

Model weights are a fixed cost. The conversation is not. As context grows, the model keeps a cache of intermediate values — the KV cache — and that cache lives in the same VRAM as the weights. It grows with the length of the conversation, and on a card that was already nearly full it is what makes a session that started fine slow to a crawl half an hour in.

This is also why advertised context windows are misleading. A model may support 256,000 tokens; whether you can use 256,000 tokens depends entirely on how much memory is left once the weights are loaded.

The good news is that the cache can be quantised too, and the effect is dramatic. Figures published in the llama.cpp project's TurboQuant discussion, for a 70B model at Q4_K_M with 34 GB free for cache: about 109,000 tokens of context at FP16, about 218,000 at Q8_0, and about 536,000 using TurboQuant's 3.25-bit scheme. Each step down roughly doubles what you can hold.

Running llama.cpp directly, the relevant flags are --cache-type-k and --cache-type-v, and flash attention has to be enabled for the value cache to be quantised at all. For most people the practical takeaway is simpler: quantising the KV cache is usually a bigger win than agonising over the weight quantisation, and it is the first thing to try when long conversations start hurting.

Mixture-of-experts changes the arithmetic — but only half of it

A great many of 2026's most interesting models are mixture-of-experts designs, often written as something like 30B-A3B: 30 billion total parameters, roughly 3 billion active for any given token. Gemma 4's 26B-A4B, gpt-oss-20b, GLM-4.7-Flash, Cohere's North Mini Code and several NVIDIA Nemotron variants all follow this pattern.

The important thing is which half of the equation it helps. You still have to hold every expert in memory, so a 30B-A3B model costs you the memory of a 30B model — around 19 to 24 GB at Q4_K_M. What you gain is speed: only the active slice is computed per token, so it generates at something closer to 3B pace.

For a homelab with one big card and no datacentre interconnect, that is an excellent trade. It is also why the roughly 30B-total, 3B-active shape became the standard target for local models this year — it is tuned almost exactly to a 24 GB card.

The flip side is the enormous MoE models. A model with 2.4 trillion total parameters and 95 billion active is still a 2.4-trillion-parameter memory problem, and no amount of sparsity brings that within reach of a desktop.