{"id":"arxiv/2609.39361","url":"https://arxiv.org/abs/2609.39361","tags":["cs.LG","math.NA"],"score":0,"title":"LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators","domain":"cs.LG","source":"arxiv","authors":["Stanislav Budzinskiy","Marian Gloser","Tolunay Yilmaz","Ying Hong Tham","Yuanyi Lin","Wenyi Fang","Fan Wu","Philipp Petersen"],"reviews":0,"subject":"cs.LG","summary":"While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel hardware-algorithm co-design in the form of mixed-precision FlashAttention. Our method accumulates key-query products and evaluates their exponentials in 8-bit formats, then adaptively identifies sensitive sub-blocks and recomputes them in 16-bit formats. We propose the specifications for a dedicated accelerator capable of executing this pipeline efficiently. Simulated experiments with Qwen3 and Gemma 3 show that…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-09-30","reactions":{}}