{"id":"arxiv/2608.30417","url":"https://arxiv.org/abs/2608.30417","tags":["cs.LG","math.AG","math.RT"],"score":0,"title":"Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention","domain":"cs.LG","source":"arxiv","authors":["Tīkun Ông","Georg Bökman"],"reviews":0,"subject":"cs.LG","summary":"We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$ can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-08-31","reactions":{}}