Universal Approximation with Deep Narrow Networks

The classical Universal Approximation Theorem certifies that the universal approximation property holds for the class of neural networks of arbitrary width. Here we consider the natural `dual' theorem for width-bounded networks of arbitrary depth, for a broad class of activation functions. In particular we show that such a result holds for polynomial activation functions, making this genuinely different to the classical case. We will then discuss some natural extensions of this result, e.g. for nowhere differentiable activation functions, or for noncompact domains.