How a tiny AI learns to imitate a giant one

The best AI models are often huge and expensive to run. Knowledge distillation is the trick that lets a small, cheap model capture much of a big model's skill — by training the small one to imitate the big one instead of learning from scratch.

Here's the clever part. A normal model is trained on hard labels: this image is 100% "dog," 0% everything else. But when you ask the big teacher model, it outputs soft labels — a full probability spread like 90% dog, 7% wolf, 3% cat. That spread encodes what the teacher considers almost right, revealing hidden relationships between categories. Researchers call this extra signal dark knowledge.

Tap to zoom

By matching those soft targets, the student learns not just the answer but the teacher's reasoning texture — often reaching accuracy far beyond what its small size would allow if trained on raw labels alone. This is how many fast, on-device models are built today: a small network trained to echo a much larger one.