{"id":"arxiv/2609.35349","url":"https://arxiv.org/abs/2609.35349","tags":["cs.LG","math.NA"],"score":0,"title":"Quasi Linear Kernel Attention with Infinite Capacity","domain":"cs.LG","source":"arxiv","authors":["Nicolaj Rux","Johannes Hertrich","Sebastian Neumayer"],"reviews":0,"subject":"cs.LG","summary":"The evaluation cost of transformers with softmax attention scales quadratically with sequence length. Kernel attention addresses this by replacing softmax with a more general kernel function. In this paper, we aim to identify kernels that retain the expressivity of attention while enabling quasi linear computation. To quantify expressivity, we introduce a capacity for each kernel, measuring the maximum sequence length for which the attention matrix can approximate the identity. A higher capacity thus indicates greater expressivity. We show that expressive kernels like softmax, Gauss, and…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-09-28","reactions":{}}