{"id":"arxiv/2605.12697","url":"https://arxiv.org/abs/2605.12697","tags":["stat.ML","cs.LG","math.PR"],"score":0,"title":"A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention","domain":"stat.ML","source":"arxiv","authors":["Tomohiro Hayase","Ryo Karakida"],"reviews":0,"subject":"stat.ML","summary":"Length-dependent logit rescaling is widely used to stabilize long-context self-attention, but existing analyses and methods suggest conflicting inverse-temperature laws for the context length $n$, ranging from $(\\log n)^{1/2}$ to $\\log n$ and $(\\log n)^2$. We provide a general theory showing that the desirable scale is determined by the gap-counting function $N_n$ of each attention row. Counting how many competitors lie within each gap from the maximum, we define an upper-tail accumulation scale and prove that it gives the critical inverse-temperature scale for softmax concentration: below…","category":"research","comments":0,"disputed":false,"division":"stat","mentions":0,"discussed":0,"published":"2026-05-12","reactions":{}}