{"id":"arxiv/2605.06826","url":"https://arxiv.org/abs/2605.06826","tags":["stat.ML","cs.IT","math.SP"],"score":0,"title":"How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models","domain":"stat.ML","source":"arxiv","authors":["Mohamed El Amine Seddik"],"reviews":0,"subject":"stat.ML","summary":"We study the spectral properties of sample covariance matrices constructed from pooled sequence representations, where token embeddings are drawn from a fixed two-class Gaussian mixture table and pooled via (fixed) attention weights. Working in the high-dimensional regime $d,V,N\\to\\infty$ with $d/V\\toδ$ and $d/N\\toγ$, we derive exact characterizations of the limiting eigenvalue distribution, outlier eigenvalues, and eigenvector alignment with the hidden signal. The bulk spectrum follows a non-Marchenko--Pastur law given by the free multiplicative convolution $κ(MP_δ\\boxtimes MP_γ)$,…","category":"research","comments":0,"disputed":false,"division":"stat","mentions":0,"discussed":0,"published":"2026-05-07","reactions":{}}