LLM quantization explained: running big models on small hardware

Quantisation stores model weights with fewer bits so large models fit on smaller GPUs and laptops. How it works, the main methods, the quality trade-offs and how to choose a format.

Quantisation stores a language model’s weights with fewer bits, such as 8 or 4 instead of 16, so the model needs much less memory and often runs faster, at the cost of a small, usually acceptable, loss of accuracy. It is the main reason capable open models can run on a single GPU, a laptop or even a phone.

If you have ever wondered how people run large models at home, the answer is mostly quantisation. It is also central to serving models cheaply in production and to fine-tuning them on limited hardware.

Why does model size matter so much?

Memory needed for weights is roughly the number of parameters multiplied by the bytes per parameter. A 7-billion-parameter model in 16-bit precision needs about 14 GB for weights alone, before the memory for processing inputs. Store each weight in 4 bits and the weights need roughly a quarter of that. That is often the difference between needing a data-centre GPU and running on a consumer one.

These are rough figures for weights only; real usage also includes activations, the KV cache and runtime overhead.

How does quantisation work?

Weights are mapped from high-precision numbers to a smaller set of values, with a scale factor so they can be approximately recovered during computation. Done naively, this damages quality. Modern methods choose the mapping carefully, often per small group of weights, and protect the weights that matter most.

What are the main quantisation methods?

What is GGUF and llama.cpp?

llama.cpp is an open-source project for running language models efficiently on ordinary hardware, including CPUs and laptops, and GGUF is its model file format, which supports several quantisation levels. It is a popular way to run open models locally for experimentation, privacy or offline use.

How much quality do you lose?

It depends on the model, the method and the bit width. 8-bit quantisation is usually close to the original; 4-bit with good methods is often acceptable for many tasks; very low bit widths degrade more noticeably. The only reliable answer is to measure on your own tasks, with the same evaluation set you use for everything else. Our guide to evaluating LLM applications shows how.

How do you quantise a model in practice?

Quantisation also matters for privacy: running a quantised open model on your own hardware means sensitive data never leaves your systems, which can simplify compliance for Indian businesses handling personal data.

When should you quantise?

For learners, quantisation is also the most practical way to experiment with capable models without expensive hardware. A quantised open model on a laptop lets you practise prompting, retrieval and evaluation end to end, then move to larger hardware only when a project actually needs it.

What are common quantisation pitfalls?

How does this fit with other efficiency techniques?

Quantisation reduces memory per parameter; mixture of experts reduces compute per token; KV-cache management and batching improve serving efficiency. See mixture of experts explained and how small language models are trained for the bigger picture.

Where can you learn this hands-on?

Program Zero covers LoRA and QLoRA fine-tuning in its LLM phase and inference optimisation, including KV-cache management and quantisation with GPTQ, AWQ and GGUF, in its advanced LLM engineering phase.

Program Zero runs live online, so you can learn this from anywhere in India: metros such as Mumbai, Delhi NCR, Kolkata and Chennai, growing cities such as Chandigarh, Bhopal, Surat and Vadodara, and smaller towns such as Dehradun, Jammu, Gwalior and Jabalpur. Every learner joins the same evening classes, at the same fee.

Frequently asked questions

Related reading