Quantization (GGUF, Q4_K_M)

The download page lists twenty versions of the same model, named like licence plates: Q4_K_M, Q5_K_S, Q8_0. The letters tell you how hard the model was squeezed to fit your hardware.

The short answer

Quantization stores the weights of a language model with fewer bits, for example as 4-bit integers instead of 32-bit floating-point numbers, which shrinks the file and can speed up inference [1]. GGUF is the file format that llama.cpp and other GGML-based tools load; besides the tensors, it carries a standardized set of metadata [2][3]. So what is Q4_K_M? A llama.cpp quantization type that stores most of the model as 4-bit "K-quants" and selected tensors with 6 bits [1][5]. Our assessment: Q4_K_M is a sensible default when memory is tight, because llama.cpp's own figures show far less quality loss than the older Q4_0 for a slightly larger file [4]. If memory allows, a 5- or 6-bit type loses even less.

How it works

1. Fewer bits per weight. The tool llama-quantize takes a GGUF file in a high-precision format such as F32 or BF16 and writes a quantized copy [1]. For Llama-3.1-8B, llama.cpp lists F16 at 16.0005 bits per weight and 14.96 GiB, Q8_0 at 8.5008 bits and 7.95 GiB, and Q4_K_M at 4.8944 bits and 4.58 GiB [1].
2. Blocks with their own scale. Quantized weights sit in blocks that share scale factors. Hugging Face describes Q4_K as super-blocks of 8 blocks with 32 weights each, with a 6-bit scale and a 6-bit minimum per block, which results in 4.5 bits per weight; Q6_K comes to 6.5625 bits [2].
3. The K and the M. "K" marks the K-quant family, and the suffix names a mix. In the llama.cpp code, Q4_K_M keeps most tensors at Q4_K but stores the output layer and parts of the attention and feed-forward layers with the 6-bit type Q6_K. Q4_K_S also keeps the output layer at Q6_K, but it upgrades fewer attention and feed-forward layers, and only to the 5-bit Q5_K [5]. That mix is why Q4_K_M averages more than 4.5 bits per weight [1][2].
4. The price in quality. llama.cpp lists how much perplexity, a measure of how surprised a model is by test text, rises for each type on Llama-3-8B: +0.4685 for the older Q4_0, +0.1754 for Q4_K_M, +0.0569 for Q5_K_M and +0.0026 for Q8_0 [4]. Smaller means closer to the original. An importance matrix (imatrix) file can reduce the loss further [1].
5. One file, everything inside. GGUF was designed for single-file deployment and mmap compatibility; a file starts with the bytes "GGUF", and the current specification describes version 3 [3]. Hugging Face lists llama.cpp, LM Studio, GPT4All and Ollama among the tools that use it [2].
6. Pick before you download. Ollama imports a GGUF file as it is and does not quantize GGUF models during import, so you choose the type when you download or convert the file [6].

A practical example

You want an 8B model on a graphics card with 8 GiB of VRAM. The Q8_0 file of Llama-3.1-8B takes 7.95 GiB and would leave almost no room for the context, while Q4_K_M takes 4.58 GiB [1]. If you only have a BF16 file, llama.cpp converts it:

./build/bin/llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M

To use the result in Ollama, write a Modelfile with the single line FROM ./model-Q4_K_M.gguf and run ollama create my-model [6]. Then watch the real memory use while the model answers a long prompt. These are llama.cpp's published figures, not measurements on our hardware.

Checklist

• Check the quantization type in the file name before you download a model [1].
• Compare the file size with your free VRAM or RAM and leave headroom for the context.
• Prefer K-quants such as Q4_K_M over the legacy Q4_0, which Hugging Face describes as not widely used any more [2][4].
• Start from a high-precision file such as F32 or BF16 when you quantize yourself [1].
• Note model, quantization type and source file, so that a result can be reproduced after the next update.