Professor Ruoyu Sun will talk about: 'Understanding and Improving LLM Training via Hessian and Spectral Analysis' 

In the first part, we investigate the approximate block-diagonal Hessian structure of neural networks. We identify the conditions under which this structure emerges and give the first rigorous proofs based on random matrix theory. From this structural perspective, we explain why Adam works far better than SGD on Transformers. Following this structural guideline, we design the memory-efficient optimizer Adam-mini; Normuon is another optimizer developed under the same principle.

 In the second part, we adopt a spectral perspective to study and refine normalization layers for neural network training. We propose a preconditioning (PC) layer, an advanced weight-centric module built with low-degree polynomial preconditioning for scalable spectral control. Theoretically, for deep linear networks, we prove that bounding each layer's singular values ensures geometric convergence of gradient descent to global minima. Empirically, PC delivers consistent efficiency gains over a standard Transformer baseline in Llama2-1B pretraining.