{"id":"arxiv/2606.10944","url":"https://arxiv.org/abs/2606.10944","tags":["cs.LG","cs.DS","math.ST","stat.ME","stat.ML"],"score":0,"title":"Express Language Modeling","domain":"cs.LG","source":"arxiv","authors":["Albert Gong","Annabelle Michael Carrell","Raaz Dwivedi","Lester Mackey"],"reviews":0,"subject":"cs.LG","summary":"We introduce a new tool, Express, for converting a non-causal attention approximation into a causal approximation with matching approximation guarantees. When combined with the state-of-the-art Thinformer approximation, Express improves upon the best known causal attention guarantees, delivering $\\log^{3/2}(n)/s$ approximation error with only $O(s)$ memory and $O(s^2 \\log^2(n))$ compression overhead for a sequence of length $n$. We pair these developments with an efficient I/O-aware Triton implementation, demonstrate substantial speedups over FlashAttention 2, and use Express to overcome…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-06-09","reactions":{}}