Fast Convex Pruning of Deep Neural Networks

Abstract

We develop a fast, tractable technique called Net-Trim for simplifying a trained neural network. The method is a convex postprocessing module, which prunes (sparsifies) a trained network layer by layer, while preserving the internal responses. We present a comprehensive analysis of Net-Trim from both the algorithmic and sample complexity standpoints, centered on a fast, scalable convex optimization program. Our analysis includes consistency results between the initial and retrained models before and after Net-Trim application and provides a sample complexity bound on the number of input samples needed to discover a network layer with sparse topology. Specifically, if there is a set of weights that uses at most s terms that can recreate the layer outputs from the layer inputs, we can find these weights from \scrO(s log N/s) samples, where N is the input size. These theoretical results are similar to those for sparse regression using the Lasso, and our analysis uses some of the same recently developed tools (namely recent results on the concentration of measure and convex analysis). Finally, we propose an algorithmic framework based on the alternating direction method of multipliers (ADMM), which allows a fast and simple implementation of Net-Trim for network pruning and compression.