{"id":"arxiv/2609.19363","url":"https://arxiv.org/abs/2609.19363","tags":["cs.LG","math.NA"],"score":0,"title":"Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention","domain":"cs.LG","source":"arxiv","authors":["Rubén Darío Guerrero"],"reviews":0,"subject":"cs.LG","summary":"The query and key projections $\\WQ,\\WK$ in attention are almost always trained by Euclidean optimizers with no geometric constraint. We constrain them to the Stiefel manifold and optimize with a Riemannian Adam carrying one scalar second moment per frame---the form of \\citet{becigneul2019}, here extended to the compact, non-Hadamard $\\St(d,r)$ with a tangent projector, step-norm cap, and polar retraction. Four propositions prove steepest descent in the embedded metric, gradient-scale independence, well-conditioning, and exact $\\mathrm{O}(d)$-equivariance. A fifth records that weight decay…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-09-16","reactions":{}}