{"id":"arxiv/2608.05136","url":"https://arxiv.org/abs/2608.05136","tags":["cs.LG","math.OC","stat.ML"],"score":0,"title":"The Loss Does Not See the Basis, but Adam Does","domain":"cs.LG","source":"arxiv","authors":["Devender Singh"],"reviews":0,"subject":"cs.LG","summary":"Gradient descent on a factored model $W = UV^\\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the difference to the gauge symmetry of the loss, its invariance under $(U, V) \\mapsto (UQ, VQ)$. Gradient flow's low-rank mechanism is available to an optimizer only if that optimizer is gauge-equivariant, a condition necessary for the transfer but not sufficient for low-rank recovery. Gradient descent, momentum, \"shared-scalar\" Adam, Muon, and Shampoo satisfy it. Adam, RMSProp, and the other coordinate-wise methods do…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-08-05","reactions":{}}