LLM Quantization

What Is LLM Quantization?

LLM quantization is the process of storing a language model's weights at lower numeric precision, for example 4-bit integers instead of 16-bit floats, so the model needs less memory and runs faster. Done well, it cuts hardware requirements by half to three quarters while losing only a small amount of output quality.

Key Takeaways

• Quantization is the main reason capable models run on laptops and single GPUs: a 4-bit version needs roughly a quarter of the memory of the 16-bit original.
• Quality loss is real but usually small down to 4-bit; below that, degradation gets noticeable, especially on reasoning and code.
• The same model name at different quantization levels can behave differently, which matters when a local model backs a coding agent.
• API users mostly never see quantization, but self-hosters choose a level on every deployment, and it is the biggest single cost lever they control.

How It Works

A model's weights are just numbers, typically trained and stored as 16-bit floating point. Quantization maps those values onto a coarser grid: 8-bit integers, 4-bit integers, or mixed schemes where sensitive layers keep more precision. Fewer bits per weight means a smaller file, less GPU memory, and faster inference, because moving weights through memory is the bottleneck when generating tokens. A model with 70 billion parameters needs about 140 GB at 16-bit but closer to 40 GB at 4-bit, which is the difference between a multi-GPU server and a single high-memory card or a well-equipped Mac.

The dominant approach is post-training quantization: take a finished model and convert it, sometimes using a small calibration dataset to decide how to round each layer with minimal damage. Popular formats in the local-model ecosystem, such as GGUF quant levels and methods like AWQ and GPTQ, are all variations on this idea, differing in how cleverly they protect the weights that matter most. The 2023 GPTQ paper showed a 175-billion-parameter GPT model quantized to 3 or 4 bits per weight in about four GPU hours with negligible accuracy loss, delivering inference speedups of roughly 3.25x on an NVIDIA A100 and 4.5x on an A6000 [1]. MIT's AWQ, paired with its TinyChat runtime, reports more than 3x speedup over the Hugging Face FP16 baseline on both desktop and mobile GPUs [2]. Some models are now trained with quantization in mind, which shrinks the quality gap further. Activations and the KV cache can be quantized too, which helps long-context workloads where the cache outgrows the weights. The tradeoff curve is not linear: 8-bit is usually indistinguishable from the original, 4-bit costs a few points on hard benchmarks, and 2-bit models often become erratic in ways that summary benchmarks understate.

Example

A developer wants a local coding assistant for a codebase that cannot leave the building. The chosen open-weight model needs about 60 GB of memory at full precision, more than the workstation's 48 GB GPU. At 4-bit it fits in roughly 20 GB with room for a long context window. In daily use, autocomplete and refactoring feel identical to the hosted original. Then the AI coding agent built on it starts fumbling multi-file edits that the hosted version handles, occasionally producing malformed tool calls. Moving to a 5-bit variant with the memory freed by a shorter context fixes the tool-call failures. The lesson the team writes down: pick the quantization level against your own agent evals, not against a leaderboard.

What People Get Wrong

The standard mistake is reading "95 percent of benchmark performance" and assuming 95 percent of practical performance. Benchmark averages hide where the loss concentrates. Quantization damage shows up disproportionately at the edges: long chains of reasoning, rare tokens, strict output formats, and agent tool calls, where one corrupted step sinks the whole trajectory. A quantized model can match the original on multiple-choice evals and still fail your JSON-emitting agent noticeably more often. Evaluate the quantized model on your actual workload, in the loop it will actually run in.

FAQ

Does quantization make an LLM less accurate? Slightly, and the amount depends on how far you push it. 8-bit is effectively lossless in practice, 4-bit trades a small amount of quality for a large memory saving, and anything below 4-bit should be treated as experimental for serious work. The University of Washington's QLoRA work is the reference point for how little 4-bit can cost: it fine-tuned a 65B-parameter model on a single 48GB GPU, and the resulting Guanaco model reached 99.3% of ChatGPT's performance after 24 hours of tuning on one GPU [3].

Do OpenAI and Anthropic quantize their models? Providers generally do not document their serving precision, and it can change behind the API. As a customer you are buying model behavior, not a bit width, which is one more reason to maintain your own evals rather than assumptions about the hosting stack.

What is the difference between quantization and distillation? Quantization compresses the same model by storing its weights more coarsely. Distillation trains a different, smaller model to imitate a bigger one. Quantization is a deployment step you can apply in an afternoon; distillation is a training project. They stack: many local models are distilled first and quantized after.

Sources

1. IST Austria / ETH Zurich (arXiv, ICLR 2023). "GPTQ quantizes a 175B GPT model to 3-4 bits in about four GPU hours with 3.25x-4.5x inference speedups." https://arxiv.org/abs/2210.17323. Accessed August 2026.
2. MIT (arXiv, MLSys 2024 Best Paper). "AWQ 4-bit quantization with TinyChat delivers over 3x speedup versus Hugging Face FP16 on desktop and mobile GPUs." https://arxiv.org/abs/2306.00978. Accessed August 2026.
3. University of Washington (arXiv). "QLoRA fine-tunes a 65B model on one 48GB GPU; Guanaco reaches 99.3% of ChatGPT's performance." https://arxiv.org/abs/2305.14314. Accessed August 2026.