The Implicit and Explicit Regularization Effects of Dropout

Part of Proceedings of the International Conference on Machine Learning 1 pre-proceedings (ICML 2020)

Bibtex »Metadata »Paper »Supplemental »

Bibtek download is not availble in the pre-proceeding


Authors

Colin Wei, Sham Kakade, Tengyu Ma

Abstract

<p>Dropout is a widely-used regularization technique, often required to obtain state-of-the-art for a number of architectures. This work observes that dropout introduces two distinct but entangled regularization effects: an explicit effect which occurs since dropout modifies the expected training objective, and an implicit effect from stochasticity in the dropout gradients. We disentangle these two effects, deriving analytic simplifications which characterize each effect in terms of the derivatives of the model and loss. Our simplified regularizers accurately capture the important aspects of dropout: we demonstrate that they can faithfully replace dropout in practice.</p>