Knowledge Distillation
Training a small model to imitate the outputs of a large one, keeping most of the quality on a target task at a fraction of the cost.
/ quick answer
In distillation, a large 'teacher' model labels many examples and a smaller 'student' model is fine-tuned on them. It works best for a narrow, well-defined task with a clear evaluation set. Check the teacher provider's terms, since some forbid using outputs to train competing models.
/ frequently asked
What is Knowledge Distillation?
In distillation, a large 'teacher' model labels many examples and a smaller 'student' model is fine-tuned on them. It works best for a narrow, well-defined task with a clear evaluation set. Check the teacher provider's terms, since some forbid using outputs to train competing models.
What is an example of Knowledge Distillation?
A team labels 20,000 product descriptions with a frontier model, fine-tunes a 7B model on them, and cuts categorisation cost by over 90% at similar accuracy on its eval set.
Why does Knowledge Distillation matter?
Training a small model to imitate the outputs of a large one, keeping most of the quality on a target task at a fraction of the cost.
What should I learn after Knowledge Distillation?
Open the related workflows and prompts further down this page: they show how this idea is applied in a real task, step by step.
/ continue exploring
Related concepts
The vocabulary this page depends on.
- →Small Language Model (SLM)
A language model with roughly 1–15B parameters, small enough to run on a laptop, phone or single GPU while handling narrow tasks well.
- →Prompt Caching
Reusing the processed form of a repeated prompt prefix so later requests are cheaper and faster.
- →Semantic Cache
A cache that returns a stored LLM answer when a new question means the same thing as an earlier one, even if worded differently.
- →Knowledge Graph
A network of entities and the relationships between them, queryable like a map.
Related workflows
Turn this into a repeatable process.
- →Cut LLM Inference Costs
A step-by-step method to reduce model spend without degrading output quality.
- →Route Requests Between Models
Send each request to the cheapest model that can handle it, escalating only when needed.
- →Personal Research Assistant Workflow
A repeatable system to research any topic deeply in under 30 minutes.
- →Cut Agent Costs by 60% Without Losing Quality
A measurable cost-reduction pass for any agent already in production.
Related tool stacks
The tools that run it in production.
- →LLM Cost Optimization Stack
The layers that keep AI spend predictable as usage grows.
- →Local AI Stack
Open-source tools to run chat, retrieval and agents entirely on your own hardware.
- →Knowledge Graph Stack
Relationship layer that maps concepts, workflows, prompts, tools, and cases.
- →AI Cost Optimization Stack
This stack provides tools and services for monitoring, analyzing, and controlling the operational costs associated with AI agent deployment and LLM usage.
Related prompts
Reusable prompts for this job.
- →Model Routing Classifier Prompt
A cheap-model prompt that labels request difficulty so it can be routed to the right model.
- →Red Team Attack Plan Prompt
Generates a tailored list of adversarial tests for your own AI feature.
Related use cases
How people apply it, and what came out.
- →SaaS Cuts AI Bill With Routing
Illustrative example: routing easy requests to a small model reduces spend while holding quality steady.
- →Support Team Replaces Wiki Sprawl With a Knowledge Graph
A support org connected policies, playbooks, tickets, and RAG answers into one system.
- →B2B Team Cuts Outbound Cost 70% With AI SDR
A Series-A team replaced 2 SDR seats with 1 orchestrator + AI, without losing pipeline.
Comparisons & alternatives
Pick between the options.
- →Local LLM vs Cloud API
Local models win on privacy and fixed cost; cloud APIs win on quality, scale and zero maintenance.
- →Reasoning Model vs Standard LLM
Reasoning models are more accurate on multi-step problems but slower and pricier; standard calls suit simple, fast tasks.
- →Vector Database vs Knowledge Graph
Similarity retrieval versus explicit relationship mapping.