Mathematical methods are developed to characterize the asymptotics of recurrent neural networks (RNN) as the number of hidden units, data samples in the sequence, hidden state updates, and training steps simultaneously grow to infinity. In the case of an RNN with a simplified weight matrix, we prove the convergence of the RNN to the solution of an infinite-dimensional ODE coupled with the fixed point of a random algebraic equation. 
The analysis requires addressing several challenges which are unique to RNNs. In typical mean-field applications (e.g., feedforward neural networks), discrete updates are of magnitude O(1/N ) and the number of updates is O(N). Therefore, the system can be represented as an Euler approximation of an appropriate ODE/PDE, which it will converge to as N → ∞. However, the RNN hidden layer updates are O(1). Therefore, RNNs cannot be represented as a discretization of an ODE/PDE and standard mean-field techniques cannot be applied. Instead, we develop a fixed point analysis for the evolution of the RNN memory state, with convergence estimates in terms of the number of update steps and the number of hidden units. The RNN hidden layer is studied as a function in a Sobolev space, whose evolution is governed by the data sequence (a Markov chain), the parameter updates, and its dependence on the RNN hidden layer at the previous time step. Due to the strong correlation between updates, a Poisson equation must be used to bound the fluctuations of the RNN around its limit equation. These mathematical methods allow us to prove a neural tangent kernel (NTK) limit for RNNs trained on data sequences as the number of data samples and size of the neural network grow to infinity.

Fluctuating hydrodynamics provides a framework for approximating density fluctuations in interacting particle systems by suitable SPDEs. The Dean-Kawasaki equation - a strongly singular SPDE - is perhaps the most basic equation of fluctuating hydrodynamics; it has been proposed in the physics literature to describe the fluctuations of the density of N diffusing weakly interacting particles in the regime of large particle numbers N. The strongly singular nature of the Dean-Kawasaki equation presents a substantial challenge for both its analysis and its rigorous mathematical justification: Besides being non-renormalizable by approaches like regularity structures, it has recently been shown to not even admit nontrivial martingale solutions.

In this talk, we give an overview of recent quantitative results for the justification of fluctuating hydrodynamics models. In particular, we give an interpretation of the Dean-Kawasaki equation as a "recipe" for accurate and efficient numerical simulations of the density fluctuations for weakly interacting diffusing particles, allowing for an error that is of arbitarily high order in the inverse particle number. 

Based on joint works with Federico Cornalba, Jonas Ingmanns, and Claudia Raithel