Rather than learning only from hard labels, the student learns from the teacher’s full probability distributions or generated answers, which carry richer information. The idea was popularized by a 2015 paper by Hinton, Vinyals and Dean.
Distillation is a main reason small, fast models have improved so quickly.