Associate Professor Zhanzing Zhu is going to talk about; 'Understanding Transformers: Token Distinguishability, Learnability, and Safety Alignment'
The underlying mechanism of Large Language Models’ (LLMs) reasoning capabilities remains a big puzzle. This talk will introduce several theoretical works from our group, which attempt to answer some fundamental questions about the Transformer in LLMs:
- Does softmax attention still work for a very long context (e.g. millions) to distinguish each token for learning? How can we scale the logit to produce a sharp attention distribution?
- Why can Transformers acquire reasoning ability through Chain-of-Thought (CoT) with fine-tuning? What is the difference between supervised fine-tuning (SFT) and reinforcement learning (RL) for fine-tuning LLMs?
- Safety alignment in LLMs is fragile in part because it is often shallow: fine-tuning mainly reshapes the model’s behavior near the first few output tokens. Why does the fine-tuning lead to such fragile safety alignment? How can we robustify it?
Further Information
Bio:
Dr. Zhanxing Zhu is an Associate Professor of Machine Learning at ECS, University of Southampton, UK. He obtained a Ph.D. degree in machine learning from the University of Edinburgh. He currently focuses on the theoretical and methodological foundations of deep learning and Physical AI. He has co-authored more than 70 papers published in top machine learning journals and conferences, including TPAMI, NeurIPS, ICML, and ICLR. He has been recognised as one of Elsevier and Stanford’s Top 2% Scientists (2020–2026) and as a 2020-2026 AI 2000 Most Influential Scholar. He has been serving as Area Chair or Senior Area Chair for NeurIPS/ICML/ICLR and as an action editor for TMLR. More information on his website: https://zhanxingzhu.github.io