What is Knowledge Distillation?

Knowledge Distillation is a machine-learning technique in which a smaller, simpler AI model (the “student”) is trained to mimic the behavior and outputs of a larger, more complex model (the “teacher”). The goal is to retain much of the teacher’s performance while making the student faster and cheaper to run.