I am a PhD student in Machine Learning, supervised by Gabriel Peyré and Pierre Ablin at Ecole Normale Supérieure, Paris.

I investigate theoretical and practical properties of Transformers.

Contact: valerie.castin (at) ens.psl.eu

News

• January 2025: Our new preprint A Unified Perspective on the Dynamics of Deep Transformers is out! Joint work with J. A. Carrillo, G. Peyré, P. Ablin. With a PDE formalism, we investigate the dynamics of tokens as they go through an infinitely deep Transformer.
• January 2025: I presented our paper How Smooth Is Attention? at the MLSP Seminar, ENS de Lyon
• September 2024: I officially started my PhD!
• June 2024: I presented a poster at the CIRM research school Frontiers in Interacting Particle Systems. Joint work with J. A. Carrillo
• April 2024: Our paper How Smooth Is Attention? was accepted at ICML 2024! We investigate the (local) Lipschitz constant of self-attention, and show that it grows with the sequence length.
• October 2023: I am starting a 3-month visit at University of Oxford, to work with José Antonio Carrillo.
• April 2023: I started my master thesis at ENS PSL with Gabriel Peyré and Pierre Ablin!