A concrete example
Llama 70B in 4-bit fits on a consumer GPU. In 16-bit, you need a datacenter. Quantization is that compression.
Why it matters
It's what makes local AI possible. Without quantization, running a big model at home is out of reach.
You'll see it in Ollama, LM Studio, and any model running locally.
Don't mix it up with
Distillation: Distillation trains a small model to imitate a more powerful one.
AI model: An AI model is the engine that reads your request and produces an answer.
Common mistakes
- Thinking 4-bit quantization gives the same quality as 16-bit.
- Forgetting that more aggressive quantization breaks reasoning models.
- Mixing up quantization (the model weights) and compute precision (the runtime).
Quick checklist
- First I check whether the word names a concept, a tool, a risk, or a metric.
- I tie it to a concrete case: Llama 70B in 4-bit fits on a consumer GPU. In 16-bit, you need a datacenter. Quantization is that compression.
- I keep the main trap in mind: Thinking 4-bit quantization gives the same quality as 16-bit.
Quick questions
What is Quantization in AI?
Quantization lowers a model's internal precision to save memory and gain speed.
Where will I run into Quantization?
You'll see it in Ollama, LM Studio, and any model running locally.
Which word should I read next?
Start with Distillation, AI model, Open source vs open weight.