Speaker Aaron Mishkin will talk about; 'Convex Analysis of Non-Convex Neural Networks'
One of the key themes in modern optimization is the boundary between convex and non-convex problems. While convex problems can often be solved efficiently, many non-convex programs are NP-Hard and formally difficult.
In this talk, we show how to break the barrier between convex and non-convex optimization by reformulating, or "lifting", neural networks into high-dimensional spaces where they become convex. These convex reformulations serve two purposes: as algorithmic tools to enable fast, global optimization for two-layer ReLU networks; and as a convex proxy to study variational properties of the original non-convex problem. In particular, we show that shallow ReLU networks are equivalent to models with simple "gated ReLU" activations, derive the set of all critical points for two-layer ReLU networks, and give the first polynomial-time algorithm for optimal neuron pruning. We conclude with extensions to ReLU networks of arbitrary depth using a novel layer-elimination argument.