Date
Mon, 24 Feb 2025
Time
14:00 - 15:00
Location
Lecture Room 3
Speaker
Claire Boyer
Organisation
Sorbonne University

Attention-based models, such as Transformer, excel across various tasks but lack a comprehensive theoretical understanding, especially regarding token-wise sparsity and internal linear representations. To address this gap, we introduce the single-location regression task, where only one token in a sequence determines the output, and its position is a latent random variable, retrievable via a linear projection of the input. To solve this task, we propose a dedicated predictor, which turns out to be a simplified version of a non-linear self-attention layer. We study its theoretical properties, by showing its asymptotic Bayes optimality and analyzing its training dynamics. In particular, despite the non-convex nature of the problem, the predictor effectively learns the underlying structure. This work highlights the capacity of attention mechanisms to handle sparse token information and internal linear structures.

This is a joint work with Pierre Marion, Gérard Biau and Raphaël Berthier

Last updated on 2 Dec 2024, 11:05am. Please contact us with feedback and comments about this page.