{"id":"arxiv/2601.16514","url":"https://arxiv.org/abs/2601.16514","tags":["cs.LG","cs.AI","math.OC"],"score":0,"title":"Finite-Time Analysis of Gradient Descent for Shallow Transformers","domain":"cs.LG","source":"arxiv","authors":["Enes Arda","Semih Cayci","Atilla Eryilmaz"],"reviews":0,"subject":"cs.LG","summary":"Understanding why Transformers perform so well remains challenging due to their non-convex optimization landscape. In this work, we analyze a shallow Transformer with $m$ independent heads trained by projected gradient descent in the kernel regime. Our analysis reveals two main findings: (i) the width required for nonasymptotic guarantees scales only logarithmically with the sample size $n$, and (ii) the optimization error is independent of the sequence length $T$. This contrasts sharply with recurrent architectures, where the optimization error can grow exponentially with $T$. The trade-off…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-01-23","reactions":{}}