I am a PhD student in Machine Learning, supervised by Gabriel Peyré and Pierre Ablin at Ecole Normale Supérieure, Paris.

I investigate theoretical and practical properties of Transformers.

I also co-organize the PhD seminar of our lab. Reach out if you would like to come present your work!

Contact: valerie.castin (at) ens.psl.eu

News

• September 2025: I was in Hamburg to present my recent works at the workshop Mathematics of Transformers. Slides
• June 2025: Our paper Opening the Black Box: Reverse-Engineering of Sparse Neural Networks has been accepted as an oral at GRETSI 2025! Joint work with Rémi Gribonval.
• January 2025: Our new preprint A Unified Perspective on the Dynamics of Deep Transformers is out! Joint work with J. A. Carrillo, G. Peyré, P. Ablin. With a PDE formalism, we investigate the dynamics of tokens as they go through an infinitely deep Transformer.
• January 2025: I presented our paper How Smooth Is Attention? at the MLSP Seminar, ENS de Lyon
• September 2024: I officially started my PhD!
• June 2024: I presented a poster at the CIRM research school Frontiers in Interacting Particle Systems. Joint work with J. A. Carrillo
• April 2024: Our paper How Smooth Is Attention? was accepted at ICML 2024! We investigate the (local) Lipschitz constant of self-attention, and show that it grows with the sequence length.
• October 2023: I am starting a 3-month visit at University of Oxford, to work with José Antonio Carrillo.
• April 2023: I started my master thesis at ENS PSL with Gabriel Peyré and Pierre Ablin!