Since their introduction in 2017, Transformers have revolutionized large language models and the broader field of deep learning. Central to this success is the ground-breaking self-attention mechanism. In this presentation, I’ll introduce a mathematical framework that casts this mechanism as a mean-field interacting particle system, revealing a desirable long-time clustering behaviour. This perspective leads to a trove of fascinating questions with unexpected connections to Kuramoto oscillators, sphere packing, Wasserstein gradient flows, and slow dynamics.
Bio: Philippe Rigollet is a Distinguished Professor of Mathematics at MIT, where he serves as Chair of the Applied Math Committee and Director of the Statistics and Data Science Center. His research spans multiple dimensions of mathematical data science, including statistics, machine learning, and optimization, with recent emphasis on optimal transport and its applications. See https://math.mit.edu/~rigollet/ for more information.
This is hosted by the AI Reading Group