{"id":"arxiv/2607.18584","url":"https://arxiv.org/abs/2607.18584","tags":["cs.LG","math.DS"],"score":0,"title":"On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers","domain":"cs.LG","source":"arxiv","authors":["Sixu Li","Thomas Jacob Maranzatto","Jan Peszek","Trevor Teolis","Semih Akkoc","Konstantin Riedl","Sennur Ulukus","Nicolás García Trillos"],"reviews":0,"subject":"cs.LG","summary":"We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles that interact dynamically through successive linear self-attention layers. We show that in embedding dimension two, for any key, query, and value matrices, the dynamics can be reformulated as a generalized Kuramoto-type model with pure second-harmonic coupling. This formulation is amenable to Watanabe--Strogatz theory which reveals the dynamics are intrinsically low-dimensional regardless of the parameter…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-07-20","reactions":{}}