Artificial Neural Networks and Kernel Methods

The random initialisation of Artificial Neural Networks (ANN) allows one to describe, in the functional space, the limit of the evolution of ANN when their width tends towards infinity. Within this limit, an ANN is initially a Gaussian process and follows, during learning, a gradient descent convoluted by a kernel called the Neural Tangent Kernel.

Connecting neural networks to the well-established theory of kernel methods allows us to understand the dynamics of neural networks, their generalization capability. In practice, it helps to select appropriate architectural features of the network to be trained. In addition, it provides new tools to address the finite size setting.