{"id":"arxiv/2605.11059","url":"https://arxiv.org/abs/2605.11059","tags":["stat.ML","cs.LG","math.PR"],"score":0,"title":"Uniform Scaling Limits in AdamW-Trained Transformers","domain":"stat.ML","source":"arxiv","authors":["William Gibson","Christoph Reisinger"],"reviews":0,"subject":"stat.ML","summary":"We study the large-depth limit of transformers trained with AdamW, by modelling the hidden-state dynamics as an interacting particle system (IPS) coupled through the attention mechanism. Under appropriate scaling of the attention heads, we prove that the joint dynamics of the hidden states and backpropagated variables converge in $L^2$, uniformly over the initial condition, to the solution of a forward--backward system of ODEs at rate $\\mathcal O(L^{-1}+L^{-1/3}H^{-1/2})$. Here, $L$ and $H$ denote the depth and number of heads of the transformer, respectively. The limiting system of ODEs can…","category":"research","comments":0,"disputed":false,"division":"stat","mentions":0,"discussed":0,"published":"2026-05-11","reactions":{}}