Core Concepts

What Is Knowledge Distillation?

Knowledge distillation is a technique in which a smaller “student” model is trained to imitate the outputs of a larger “teacher” model. The student ends up much cheaper to run while retaining a good deal of the teacher’s capability. Distillation is widely used to make powerful models practical for phones and other constrained devices.