Workshop on the Mathematics of Transformers (Aug 2026)

Talk 1: Anisotropic interaction in self-attention layers
by Samira Kabri

Abstract: In recent years, analysing transformer architectures from the perspective of interacting particle systems as proposed by Sander et al. (2022) has gained strong interest. The work by Geshkovski et al. (2024) connects the mean-field dynamics induced by self-attention layers to a time-continuous evolution of probability measures on a high dimensional sphere. In particular, it analyses the long-time clustering behaviour under isotropic interaction, which corresponds to a fixed choice of the so-called key, query and value parameters. In this talk, we relax the assumptions on the parameters such that the interaction between particles becomes anisotropic. We discuss how anisotropic interaction influences the structure of the self-attention layer and the stationary states of the dynamics.

Talk 2: Diverse Dynamical Behaviors in Deep Linear Transformers
by Thomas Jacob Maranzatto and Jan Peszek

Abstract: In this talk, we discuss recent work on the dynamics arising in linearized transformers when the token embedding space is two-dimensional. In contrast to much of the recent literature, we do not rely on the gradient flow framework. Instead, we identify a lower-dimensional invariant manifold and derive a reduced state-space description that allows us to analyze token dynamics beyond consensus and clustering. We discuss this reduction and its connection to the Kuramoto model, and present examples of token convergence, clustering, cycling, and bifurcations across different parameter regimes.
This talk is based on the paper "On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers".

Talk 3: Hardmax attention dynamics
by Domènec Ruiz-Balet

Abstract: In this talk we will discuss some theoretical aspects of hardmax attention models from a PDE viewpoint.

Talk 5: Transformers for Operator Learning on Probability Measures: From Theory to Scientific Workflows
by Edoardo Calvello

Abstract: Attention mechanisms, the core of transformer architectures that power modern large language models, have been widely successful at modeling nonlocal correlations in data. Motivated by recent formulations of attention as a measure-to-measure mapping, in this talk we show how the mathematical properties of the attention mechanism enable the design of neural architectures approximating nonlinear operators on probability measures. We leverage this perspective to design novel methods for nonlinear data assimilation, surpassing the performance of ensemble Kalman method baselines. Starting from a universal approximation theorem we obtain for the conditioning operator on densities, we discuss key considerations and open questions towards the development of scalable transformer-based Bayesian inference workflows.

Talk 6: Uniform Scaling Limits in AdamW trained Transformers
by William Gibson

Abstract: We study the large-depth limit of transformers trained with AdamW, by modelling the hidden-state dynamics as an interacting particle system (IPS) coupled through the attention mechanism.
Under appropriate scaling of the attention heads, we prove that the joint dynamics of the hidden states and backpropagated variables converge in $L^2$, uniformly over the initial condition, to the solution of a forward–backward system of ODEs at rate $\mathcal O(L^{-1}+L^{-1/3}H^{-1/2})$. Here, $L$ and $H$ denote the depth and number of heads of the transformer, respectively. The limiting system of ODEs can be identified with a McKean–Vlasov ODE (MVODE) when the attention heads do not incorporate causal masking. By using the flow maps associated with this MVODE and applying concentration of measure techniques, we obtain bounds on the difference between the discrete and continuous models that are uniform over compact sets of initial conditions. As this is achieved without resorting to a covering argument, the constants in our bounds are independent of the number of tokens. Furthermore, under a suitable adaptation to AdamW, the bounds become independent of the token embedding dimension.