An Overview Of Knowledge Distillation Techniques
Knowledge distillation is a technique for transferring knowledge from a large, complex model to a smaller, more efficient one.
Areas of application
- Natural Language Processing
- Computer Vision
- Recommendation Systems
- Robotics and Control
Example
For instance, a large language model can be used to train a smaller model on the same task, with the goal of achieving similar performance while reducing the computational requirements.