{"id":"arxiv/2609.22990","url":"https://arxiv.org/abs/2609.22990","tags":["cs.LG","math.SP","stat.ML"],"score":0,"title":"On attention heads and bilinear forms","domain":"cs.LG","source":"arxiv","authors":["Andrew O'Desky"],"reviews":0,"subject":"cs.LG","summary":"We study the symmetric and antisymmetric parts of bilinear forms in the attention heads of trained large language models. We introduce an orthogonally invariant profile map from real bilinear forms to a three-dimensional simplex and observe that profiles of trained bilinear forms accumulate near profiles of rank-one bilinear forms. We prove that the symmetric part of a bilinear form in an attention head is the sum of a hyperbolic form and a zero form for a Zariski-dense subset of query-key matrices.","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-09-19","reactions":{}}