Local AI on your PC: how much VRAM you need

Sooner or later, every forum thread about local AI arrives at the same question: 12 or 16 gigabytes? The honest answer isn't a shopping list but a sum that fits on a beer mat.

The short answer

How much VRAM do you need for a local language model? Add up three things: the weights (parameters × bits per weight ÷ 8), the KV cache for the context you want, and a reserve for the runtime. VRAM is the graphics card's own memory. The context is the text a model can take into account at once, counted in tokens, the word pieces a model reads and writes. An 8-billion-parameter model in the Q4_K_M format takes about 5 GB, and 8,192 tokens of context add about 1.2 GB [17][18][19]. With a reserve of 1 GB, 8 GB of VRAM is just enough, and 12 GB leaves room for a longer context. If a model doesn't fit, Ollama and llama.cpp move part of it into system memory. That works, but every generated token then takes longer [2][13]. All facts are as of 25 Oct 2026. We quote no speeds, because we haven't measured any for this guide.

How it works

1. Three things share the memory. First come the weights, the learned numbers in the model file. llama.cpp loads them completely, which is why its documentation says memory and disk requirements are the same [7]. Second is the KV cache, which holds intermediate results for every token in the context and therefore grows with the context length [13]. Third is everything else: buffers of the runtime and, if the card also drives your monitor, the desktop. Nobody publishes a fixed number for that part. llama.cpp's automatic fitting tries to keep 1,024 MiB free on each GPU by default [8]. We use that as our planning reserve. It is a target value, not a measured overhead.
2. Weights: parameters × bits ÷ 8. "8B" stands for 8 billion parameters. At 16-bit precision, each one takes 2 bytes, which is why NVIDIA puts a 7-billion-parameter model at roughly 14 GB [13]. Quantization stores the weights with fewer bits. llama.cpp's own table for Llama 3.1 8B shows the effect: 14.96 GiB at F16, 7.95 GiB at Q8_0 and 4.58 GiB at Q4_K_M [7]. The formula holds up against real files. Qwen3-8B has 8.2 billion parameters [17]. With the 4.89 bits per weight that llama.cpp's table gives for Q4_K_M, 8.2 billion × 4.89 ÷ 8 makes 5.0 GB, and Qwen's official Q4_K_M file weighs 5.03 GB [7][19].
3. What names like Q4_K_M tell you. GGUF is the model file format of llama.cpp, and Hugging Face lists Ollama among the tools that use it [11]. The type names describe how the weights are stored. Q8_0 keeps blocks of 32 weights at 8 bits. The k-quants Q6_K, Q5_K and Q4_K group weights in super-blocks and come to 6.5625, 5.5 and 4.5 bits per weight [11]. A file labelled Q4_K_M averages more than its type: llama.cpp's table gives 4.89 bits per weight for Llama 3.1 8B, and 4.67 for Q4_K_S [7]. So parts of such a file use more bits than the name suggests. For planning, take the average from the table, not the nominal value of the type.
4. Fewer bits cost precision. Quantization lowers the precision of the stored weights; that's the price of the smaller file. How much quality you lose depends on the model and the task. llama.cpp's table lists sizes and speeds, but no quality measure [7]. Qwen's official GGUF files range from Q4_K_M to Q8_0 [19]. Our suggestion: try two or three variants from that range with your own prompts before you settle on one.
5. The KV cache, the part people forget. While a model writes, it keeps a key and a value for every token of its context, in every layer. NVIDIA gives the size per token as 2 × layers × (heads × head dimension) × bytes per value [13]. Many current models use grouped-query attention (GQA), which stores fewer key and value heads and so shrinks the cache [13]. That's why you take the number of key-value heads from the model's config.json, not the number of attention heads: num_hidden_layers, num_key_value_heads and head_dim. Qwen3-8B, for example, has 36 layers, 8 key-value heads and a head dimension of 128 [18]. At 2 bytes per value (f16), that makes 2 × 36 × 8 × 128 × 2 = 147,456 bytes per token, or 1.21 GB for 8,192 tokens and 4.83 GB for 32,768 tokens. The formula fits classic transformer models like this one. For other designs, trust the memory figure your runtime shows.
6. Context length: check the default. As of 25 Oct 2026, Ollama's documentation sets the default context by VRAM: 4k tokens below 24 GiB, 32k from 24 to 48 GiB and 256k from 48 GiB [2]. Its FAQ still names 4,096 tokens as the default [3], so check what you actually got with ollama ps (see below). For agents, web search and coding tools, Ollama recommends at least 64,000 tokens [2]. You change the length with the slider in the app, with the environment variable OLLAMA_CONTEXT_LENGTH or per request with num_ctx [2][3]. llama.cpp's server takes the context size from the model unless you set one [8]. Its fit option, on by default, adjusts unset values until everything fits into memory, but it doesn't go below 4,096 tokens of context [8].
7. A smaller cache. Both tools can store the cache with fewer bits. In Ollama, OLLAMA_KV_CACHE_TYPE=q8_0 needs about half the memory of the default f16, with a very small loss of precision; q4_0 needs about a quarter, with a small to medium loss [3]. This works with Flash Attention enabled, which Ollama uses automatically when the backend and devices support it, and the setting applies to all models at once [3]. In llama.cpp, --cache-type-k and --cache-type-v set the types; the default is f16 [8].
8. When the model doesn't fit. Both tools can split a model between VRAM and system memory [3][9]. ollama ps then shows a split such as 48%/52% CPU/GPU instead of 100% GPU [3]. In llama.cpp, -ngl sets how many layers go into VRAM [8]. Generating text is memory-bound: the pace is set by how fast data moves from memory, not by the arithmetic [13]. Every layer in system memory therefore costs time on every token, and Ollama recommends avoiding offloading to the CPU for best performance [2]. The offloaded part also needs free system memory; our page on RAM explains the basics.
9. NVIDIA, AMD, Apple or just the processor. Ollama supports NVIDIA cards from compute capability 5.0 with driver 550 or newer; cards with compute capability 5.0 to 6.2 need driver 570 or newer [1]. On Linux, AMD cards need the ROCm v7 driver, and the supported list includes models from the Radeon RX 9000, 7000 and 6000 series [1]. On Windows, Ollama's ROCm list names only seven Radeon RX 7000 models and the Radeon PRO W7500 to W7900 [1]. Vulkan adds support for more GPUs on Windows and Linux and is enabled by default when its backend is installed [1]. On Apple devices, Ollama uses Metal [1]. On a Mac with Apple silicon, the processor and the GPU share the system memory [27], and Metal reports for each device how much memory the GPU can allocate without losing performance (recommendedMaxWorkingSetSize) [14]. So plan with that value, not with the full RAM. Without a GPU, llama.cpp uses AVX, AVX2, AVX-512 and AMX on x86 processors and ARM NEON on Apple silicon [9]. Our assessment: that works for small models, but since generation is memory-bound, expect fewer tokens per second than on a graphics card, and measure them [13].
10. Mixture-of-experts models count in full. Some models use only part of their weights for each token. Qwen3-30B-A3B has 30.5 billion parameters, of which 3.3 billion are active [22]. For memory, the total counts, because all experts have to be loaded [12]. Its official Q4_K_M file accordingly weighs 18.6 GB [24]. The upside, according to Hugging Face: the computing effort per token resembles that of a much smaller model [12].
11. Local stays local, if you let it. Ollama says it doesn't see your prompts or data when you run models locally [3]. Its cloud models, however, run on Ollama's servers; in the app and CLI they carry names like gemma4:cloud and require a sign-in [5]. If nothing should leave your machine, set OLLAMA_NO_CLOUD=1 or disable_ollama_cloud in ~/.ollama/server.json [3]. The local server listens on 127.0.0.1, port 11434 [3]. According to Ollama's documentation, this local API doesn't require authentication [28], so keep it there. In Docker, that means adding 127.0.0.1 to the port mapping; the documentation's example publishes a plain 11434:11434 [6].

What fits where: four models compared

Here is the sum for one model family, with the official Q4_K_M files and an f16 cache. The layer and head counts come from each model's config.json, the file sizes from the official download lists, and all four models have a head dimension of 128 [18][20][23][25]. The totals are our calculation, not a measurement.

Read as VRAM sizes (our reading): with 8 GB, only the 8B model fits, at 8,192 tokens and barely. 12 GB takes the 8B model with 32,768 tokens (10.9 GB) or the 14B model with 8,192 tokens. 16 GB holds the 14B model at 32,768 tokens only just (15.4 GB). With 24 GB, the mixture-of-experts model fits even at 32,768 tokens (22.9 GB), the dense 32B model at 8,192 tokens. Hugging Face shows file sizes in GB (10⁹ bytes), while llama.cpp's table and many tools count in GiB (2³⁰ bytes), about 7 % more per unit [16]. This table uses GB throughout.

We do the same sum for Byte, our AI mascot: its language model has 9 billion parameters and runs 4-bit quantized with llama.cpp on a separate laptop in our network, as our LLM page describes. This guide contains no measurement from that machine.

Measure instead of guessing

Load the model and look at what Ollama reports. ollama ps lists each loaded model with its size, the split between processor and graphics card, the allocated context and how long it stays in memory [2][3]. This is the example from Ollama's documentation:

NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL
gemma4:latest    c6eb396dbd59    9.6 GB    100% GPU     131072     2 minutes from now

You want to see 100% GPU. If a CPU share shows up, reduce the context, choose a smaller quantization or a smaller model.

For speed, Ollama's API returns usage fields with every answer, in the last chunk when it streams: eval_count is the number of generated tokens, eval_duration the time that took, in nanoseconds [4]. Tokens per second = eval_count ÷ eval_duration × 1,000,000,000. A request like the one in Ollama's FAQ is enough; replace llama3.2 with your model [3]:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "options": {
    "num_ctx": 4096
  }
}'

To show the arithmetic: the sample response in Ollama's documentation lists 18 tokens in 52,479,709 nanoseconds, about 343 tokens per second [4]. That sample comes from unknown hardware and says nothing about your PC. With llama.cpp, llama-bench measures prompt processing (pp 512) and text generation (tg 128) by default, repeats each test five times and reports the mean with its standard deviation [10]. Write down the date, model file, quantization, context and driver version with every result.

For power, measure the whole PC at the wall with an energy meter, once at idle and once while the model is writing. By default, Ollama keeps a model in memory for 5 minutes after the last request [3], so wait until ollama ps no longer lists it before you measure idle. On NVIDIA cards, nvidia-smi also shows the board's power draw and memory use; -d POWER,MEMORY limits the output to those values, and -l 1 repeats it every second [15]:

nvidia-smi -q -d POWER,MEMORY -l 1

The card's value leaves out the processor, the RAM and the losses of the power supply, so use the wall meter for costs. The arithmetic is the one from our energy cost guide: kWh = watts × hours ÷ 1,000, and cost = kWh × your unit price. Assumed values, not measurements: 300 W at the wall for two hours a day makes 0.6 kWh a day and 219 kWh a year, which is €87.60 at the example price of €0.40 per kWh used in that guide.

Before you buy

We don't recommend a specific card; these criteria decide. First, the models and the context you actually want: take the sum from the table or from your own calculation, plus the reserve. Second, support for your exact card and operating system in Ollama's list, especially for AMD on Windows [1]. Third, power supply, connectors and space in the case; our GPU upgrade guide shows how to check them. Fourth, the price per gigabyte of VRAM on the day you buy, noted with the date. We list no prices.

Checklist

• Look up layers, key-value heads and head dimension in the model's config.json.
• Calculate the weights as parameters × bits per weight ÷ 8, with the average from llama.cpp's table.
• Add the KV cache for your context length and a reserve of about 1 GB.
• Set the context length explicitly instead of trusting the default.
• Run ollama ps and check that PROCESSOR shows 100% GPU.
• Measure tokens per second from eval_count and eval_duration, and write down the date.
• Measure power at the wall at idle and under load, and work out the cost with your own unit price.
• Set OLLAMA_NO_CLOUD=1 if nothing should leave your PC, and keep port 11434 on 127.0.0.1.
• Check the vendor list in Ollama's GPU documentation before you buy a card.

Related pages: Large language model / LLM; GPU upgrade: PSU, space and power connectors; Memory / RAM; Calculating homelab energy costs; Containers and OCI.