Limiting spectral distributions of random matrices arising in neural networks

We study the distribution of eigenvalues of kernel random matrices where each element is the empirical covariance between the feature map evaluations of a random fully-connected neural network. We show that, under mild assumptions on the non-linear activation function, namely Lipschitz continuity and measurability, the limiting spectral distribution can be written as successive free multiplicative convolutions between the Marchenko-Pastur law and a nonrandom measure specific to the neural network. The latter has no known analytical expression but can be simulated empirically, separately from the random matrices of interest.