I am a PhD student in Machine Learning, supervised by Gabriel Peyré and Pierre Ablin at Ecole Normale Supérieure, Paris. I am currently a research scientist intern at Karavela in Paris.

In my PhD, I investigate theoretical and practical properties of Transformers, in particular large language models. At Karavela, I use LLMs to better understand functional brain MRI data.

Contact: valerie.castin (at) dma.ens.fr

News

• November 2026: I presented KV-Lingo at the Workshop Numerical Stability in Deep Learning at RICAM, Linz!
• October 2026: I presented KV-Lingo at the seminar of the INRIA team MIND. I also started my internship at Karavela!
• September 2026: Our new preprint KV-Lingo: Learning KV-Cache Translators With Distillation is out! We train KV-cache translators that enable efficient communication between language models. Work done during my internship at Apple with Keitaro Sakamoto, Anastasiia Filippova, João Monteiro, Marco Cuturi and Pierre Ablin.
• July 2026: I have presented a poster about BaLoRA at ICML 2026 in Seoul!
• June 2026: I was in Saint-Malo to present a poster about BaLoRA at Curves and Surfaces 2026
• June 2026: Our paper A Unified Perspective on the Dynamics of Deep Transformers was accepted in the journal Foundations of Computational Mathematics!
• May 2026: I have started an internship in the Paris Apple MLR team!
• March 2026: I was awarded the Best Student Talk Prize (prix Dodu) for my presentation of BaLoRA at SMAI MODE 2026 in Nice!
• January 2026: I was in Alicante to present my work at the RSME Symposium PDEs, Inverse Problems and Machine Learning. Slides
• September 2025: I was in Hamburg to present my recent works at the workshop Mathematics of Transformers. Slides
• June 2025: Our paper Opening the Black Box: Reverse-Engineering of Sparse Neural Networks has been accepted as an oral at GRETSI 2025! Joint work with Rémi Gribonval.
• January 2025: Our new preprint A Unified Perspective on the Dynamics of Deep Transformers is out! Joint work with J. A. Carrillo, G. Peyré, P. Ablin. With a PDE formalism, we investigate the dynamics of tokens as they go through an infinitely deep Transformer.
• January 2025: I presented our paper How Smooth Is Attention? at the MLSP Seminar, ENS de Lyon
• September 2024: I officially started my PhD!
• June 2024: I presented a poster at the CIRM research school Frontiers in Interacting Particle Systems. Joint work with J. A. Carrillo
• April 2024: Our paper How Smooth Is Attention? was accepted at ICML 2024! We investigate the (local) Lipschitz constant of self-attention, and show that it grows with the sequence length.
• October 2023: I am starting a 3-month visit at University of Oxford, to work with José Antonio Carrillo.
• April 2023: I started my master thesis at ENS PSL with Gabriel Peyré and Pierre Ablin!