Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Abstract

This paper introduces and evaluates the idea described in “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity”, and reports empirical results that helped shape subsequent work in moe, scaling.