Model Distillation

What Is Model Distillation?

Distillation runs the large teacher model to generate outputs, then trains a compact student to imitate them. The student keeps much of the teacher's capability on the target task while being far cheaper to run. It is how teams ship task-specific models that approach frontier quality without frontier inference cost. The limit is that a student rarely generalizes beyond the distilled task as well as the teacher.