{"id":"arxiv/2610.19358","url":"https://arxiv.org/abs/2610.19358","tags":["cs.LG","cs.AI","math.ST"],"score":0,"title":"Asymptotic Behavior of Transformers","domain":"cs.LG","source":"arxiv","authors":["Katy Blumer","Da-Cheng Juan","Jon Kleinberg","Ravi Kumar","Cyrus Rashtchian","Andrew Tomkins"],"reviews":0,"subject":"cs.LG","summary":"In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each transformer has a…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-10-14","reactions":{}}