What is model distillation in AI?
Quick answer
Model distillation is a way to make a small AI model by training it to imitate a larger one. The big "teacher" model answers lots of examples, and the small "student" model learns to give similar answers. The result is faster and cheaper to run, usually with some loss on the hardest tasks.
Last updated
Take the free module →Module 6 of the free AI Agents 101 course, about 9 minutes. No account needed.
Updated · By Robert Breen
Why it matters for a small business
The idea was popularized in a 2015 research paper by Geoffrey Hinton and colleagues, and it is now common. It is one of the methods behind the small, cheap models with names like mini, flash or haiku that handle most routine business tasks well. Vendors don't always say exactly how a given model was made.
For a business, the point is cost and speed. A distilled or otherwise small model can answer an FAQ chatbot or sort emails at a fraction of the price of a frontier model. It also explains a terms-of-service detail: OpenAI's terms, for example, don't allow using its outputs to build competing models.
In a real lesson: Your First Voice Agent in ElevenLabs: An After-Hours Receptionist
In Your First Voice Agent in ElevenLabs, you click the model under LLM, open the list and pick a fast one for Riley, the Cedar Creek Plumbing receptionist. The lesson explains why: the LLM is charged on top of call minutes and depends on the model, and on a phone call the answer has to come back quickly.
Small, fast models are exactly what distillation and similar methods are used to make. You don't need to know how a model was made to choose it. Test it with real caller questions and read the transcript.
Try this lesson free or read the step-by-step guide.
Common confusions
Distillation vs fine-tuning
Fine-tuning adjusts a model with example data for a job. Distillation is training a smaller model to copy a bigger model's behavior.
Distillation vs summarization
Some people say "distill" for summarizing a document. In AI, model distillation means building a smaller model.
Tips
- Try the small model first for simple, repeated jobs; switch up only if it fails your tests.
- Re-test when a vendor swaps in a new small model.
- Read the terms before using one AI's output to train another model.
Related terms
More AI basics terms
Where you use it: free lessons
- Your First Voice Agent in ElevenLabs: An After-Hours Receptionist (ElevenLabs Agents, 9 min)
- Add an AI Chatbot to Your Website with Lovable AI (Lovable, 8 min)
Frequently asked questions
- Are distilled models worse?
- They are usually a little weaker on hard reasoning but close on everyday tasks, and much faster and cheaper.
- Can I distill a model myself?
- Developers can, with enough data and computing, but check the source model's license or terms first. Most businesses simply pick an existing small model.
- Is a mini model distilled?
- Sometimes. Vendors don't always say how a small model was made; distillation is one common method.