On Teacher Hacking in Language Model Distillation

Daniil Tiapkin , Daniele Calandriello , Johan Ferret , Sarah Perrin , Nino Vieillard , Alexandre Ramé , Mathieu Blondel