DistilBERT, a distilled version of BERT

Abstract

This paper introduces and evaluates the idea described in “DistilBERT, a distilled version of BERT”, and reports empirical results that helped shape subsequent work in distillation, nlp.