Model Distillation

What Is Model Distillation?

Model distillation is the technique of training a smaller student model to reproduce the behavior of a larger teacher model, using the teacher's outputs as training signal. The student keeps most of the teacher's capability on the targeted tasks while running faster and cheaper, which makes it a standard tool for cutting inference cost. The canonical early result is DistilBERT, which Hugging Face reported in 2019 as 40% smaller and 60% faster than BERT while retaining 97% of its language understanding [1].

Key Takeaways

• Distillation transfers capability downward: a big model generates the training signal, a small model learns to imitate it.
• The student excels inside the distribution it was trained on and degrades outside it, so distilled models suit narrow, high-volume tasks.
• Most small models you can buy or download today were trained partly on outputs from larger models; distillation is how cheap models got good.
• For agent teams, distillation is a cost play: replace a frontier model with a distilled one on the routine slice of your traffic, verified by evals.

How It Works

Classic distillation trains the student to match the teacher's full probability distribution over next tokens, the soft targets, which carry more information than the bare correct answer because they encode what the teacher considered plausible. That requires access to the teacher's internals, so it is mostly used by labs shrinking their own models. Google DeepMind trained the Gemma 2 2B and 9B models this way, using distillation in place of plain next-token prediction, and reports they compete with models two to three times bigger [2]. The variant practitioners actually touch is sequence-level or black-box distillation: generate a large set of prompts representative of your workload, collect the teacher's responses through its API, and fine-tune the student on those prompt-response pairs. This is synthetic data generation with a specific purpose, and it is why provider terms of service often restrict using outputs to train competing models.

The economics drive everything. A frontier model may cost an order of magnitude more per token than a small model and respond several times slower. If a distilled student holds, say, most of the teacher's quality on your specific task, you can route the bulk of production traffic to it and reserve the teacher for hard cases. The gap can even invert on narrow tasks: Google's 2023 Distilling Step-by-Step method let a fine-tuned 770M-parameter T5 outperform a few-shot prompted 540B-parameter PaLM, a model roughly 700 times larger, while using only 80% of the available benchmark data [3]. The catch is scope. The student did not learn to reason broadly; it learned the teacher's behavior over the training distribution. Shift the task and quality falls off faster than it would with the teacher, which is why distillation pairs naturally with evals that define the boundary of trust. Related but distinct: quantization shrinks the same model's numeric precision, while distillation produces a genuinely different, smaller model.

Example

A support-tooling team uses a frontier model to triage inbound tickets: classify the issue, extract product area and severity, draft a first response. Volume grows to two hundred thousand tickets a month, and the model bill becomes the largest line item in the product's budget. The team samples fifty thousand historical tickets, runs them through the frontier model, and fine-tunes a small open-weight model on the resulting pairs. On their eval set the distilled model matches the teacher's classification accuracy within a couple of points and writes acceptable drafts for common issue types. They ship it behind a confidence threshold: the student handles the routine majority, anything it flags as uncertain escalates to the frontier model. Inference spend drops to a fraction of the previous bill with no measurable change in customer-facing quality.

What People Get Wrong

The recurring error is expecting the student to inherit the teacher's general intelligence. Distillation copies behavior, not understanding. A student distilled on triage transcripts will triage well and reason poorly about anything else, and the gap is easy to miss because the student sounds like the teacher, fluent and confident, even when it is out of its depth. Teams that skip boundary testing discover this in production, when a novel input gets a polished, wrong answer. Define the task distribution, eval against it, and route out-of-distribution traffic to the bigger model.

FAQ

What is the difference between model distillation and fine-tuning? Fine-tuning is the training mechanism; distillation is one reason to use it. In distillation the training data comes from a larger model's outputs, so the goal is imitation and compression. Ordinary fine-tuning uses human-written or curated data to specialize a model, with no teacher involved.

Is distillation the same as quantization? No. Quantization keeps the same model and lowers the numeric precision of its weights, trading a little quality for speed and memory. Distillation trains a separate, smaller model. Teams often stack them: distill first, then quantize the student for deployment.

Can I distill a commercial model like GPT or Claude? Technically the black-box recipe works on any API, but most commercial providers prohibit using their outputs to train competing models. Check the terms before building a distillation pipeline on a third-party API; open-weight teachers avoid the problem entirely.

Sources

1. Hugging Face (arXiv, NeurIPS 2019 workshop). "DistilBERT is 40% smaller and 60% faster than BERT while retaining 97% of its language understanding." https://arxiv.org/abs/1910.01108. Accessed August 2026.
2. Google DeepMind (arXiv). "Gemma 2 2B and 9B were trained with knowledge distillation and compete with models 2-3 times bigger." https://arxiv.org/abs/2408.00118. Accessed August 2026.
3. Google Research (arXiv, ACL 2023 Findings). "Distilling Step-by-Step lets a 770M T5 outperform a 540B few-shot PaLM using 80% of benchmark data." https://arxiv.org/abs/2305.02301. Accessed August 2026.