Linear Mode Connectivity and the Lottery Ticket Hypothesis

Part of Proceedings of the International Conference on Machine Learning 1 pre-proceedings (ICML 2020)

Bibtex »Metadata »Paper »Supplemental »

Bibtek download is not availble in the pre-proceeding


Authors

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, Michael Carbin

Abstract

<p>We introduce "instability analysis," which assesses whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise. We find that standard vision models become stable in this way early in training. From then on, the outcome of optimization is determined to within a linearly connected region.</p> <p>We use instability to study iterative magnitude pruning (IMP), the procedure used by work on the lottery ticket hypothesis to identify subnetworks that could have trained to full accuracy from initialization. We find that these subnetworks only reach full accuracy when they are stable, which either occurs at initialization for small-scale settings (MNIST) or early in training for large-scale settings (Resnet-50 and Inception-v3 on ImageNet).</p>