Oberseminar "Mathematik des Maschinellen Lernens und Angewandte Analysis" - M.Sc. Albert Alcalde

We study unnormalized Transformers with self-attention and affine feed-forward layers in a continuous-depth, mean-field setting, focusing on a Gaussian regime in which the dynamics reduce to coupled equations for the mean and covariance. This reduction yields a simple and analytically tractable model that connects Transformer dynamics with classical ideas from control theory and stability analysis. We derive general qualitative properties of the resulting system, including basic controllability features, convergence and instability regimes, and the dependence of long-time behavior on parameter choices. Numerical experiments further suggest that Gaussian statistics may persist across several layers in trained encoder Transformers, indicating that this framework can serve as a simplified setting for exploring stability-related phenomena in Transformer models.