Knowledge Distillation and Model Compression

• Distilling the Knowledge in a Neural Network (2015)
• TinyBERT: Distilling BERT for Natural Language Understanding (2019)
• DistilBERT: A Distilled Version of BERT (2019)