{"id":"arxiv/2605.29351","url":"https://arxiv.org/abs/2605.29351","tags":["cs.LG","math.DS","stat.ML"],"score":0,"title":"Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics","domain":"cs.LG","source":"arxiv","authors":["Matthew Smart","Soumya Ganguly","Nilava Metya","Alexandre V. Morozov","Anirvan M. Sengupta"],"reviews":0,"subject":"cs.LG","summary":"We study minimal attention-only transformers under all-token corruption and show they admit a two-stage empirical Bayes interpretation. A single attention step computes a kernel-weighted posterior mean with respect to the empirical distribution defined by the context. Depth refines this distribution through particle dynamics (Stage 1), while a long-range skip-connection carries the noisy input as a query for posterior inference (Stage 2), revealing distinct statistical roles for depth and attention residuals. The framework isolates a minimal setting in which the context itself induces a…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-28","reactions":{}}