Express Language Modeling

Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi, Lester Mackey

arXivarxiv/2606.10944Jun 9, 2026cs.LGscore 0

We introduce a new tool, Express, for converting a non-causal attention approximation into a causal approximation with matching approximation guarantees. When combined with the state-of-the-art Thinformer approximation, Express improves upon the best known causal attention guarantees, delivering $\log^{3/2}(n)/s$ approximation error with only $O(s)$ memory and $O(s^2 \log^2(n))$ compression overhead for a sequence of length $n$. We pair these developments with an efficient I/O-aware Triton implementation, demonstrate substantial speedups over FlashAttention 2, and use Express to overcome…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2606.10944
curl -X POST /api/v1/records/arxiv/2606.10944/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2606.10944/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'