I am a PhD student in Machine Learning, supervised by Gabriel Peyré and Pierre Ablin at Ecole Normale Supérieure, Paris. I am currently a PhD intern at Apple Machine Learning Research in Paris.

I investigate theoretical and practical properties of Transformers.

Contact: valerie.castin (at) dma.ens.fr

News

• June 2026: I was in Saint-Malo to present a poster about BaLoRA at Curves and Surfaces 2026
• June 2026: Our paper A Unified Perspective on the Dynamics of Deep Transformers was accepted in the journal Foundations of Computational Mathematics!
• May 2026: I have started an internship in the Paris Apple MLR team!
• March 2026: I was awarded the Best Student Talk Prize (prix Dodu) for my presentation of BaLoRA at SMAI MODE 2026 in Nice!
• January 2026: I was in Alicante to present my work at the RSME Symposium PDEs, Inverse Problems and Machine Learning. Slides
• September 2025: I was in Hamburg to present my recent works at the workshop Mathematics of Transformers. Slides
• June 2025: Our paper Opening the Black Box: Reverse-Engineering of Sparse Neural Networks has been accepted as an oral at GRETSI 2025! Joint work with Rémi Gribonval.
• January 2025: Our new preprint A Unified Perspective on the Dynamics of Deep Transformers is out! Joint work with J. A. Carrillo, G. Peyré, P. Ablin. With a PDE formalism, we investigate the dynamics of tokens as they go through an infinitely deep Transformer.
• January 2025: I presented our paper How Smooth Is Attention? at the MLSP Seminar, ENS de Lyon
• September 2024: I officially started my PhD!
• June 2024: I presented a poster at the CIRM research school Frontiers in Interacting Particle Systems. Joint work with J. A. Carrillo
• April 2024: Our paper How Smooth Is Attention? was accepted at ICML 2024! We investigate the (local) Lipschitz constant of self-attention, and show that it grows with the sequence length.
• October 2023: I am starting a 3-month visit at University of Oxford, to work with José Antonio Carrillo.
• April 2023: I started my master thesis at ENS PSL with Gabriel Peyré and Pierre Ablin!