Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization

Xuheng Li, Qiwei Di, Yuan Cao, Quanquan Gu

arXivarxiv/2610.02798Oct 2, 2026cs.LGscore 0

The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this transformation in feature learning remain poorly understood. In this work, we investigate this question through a tractable factual-recall model, where a fact maps each subject-relation pair to an answer, and a linear transformer learns the subject- and relation-dependent information required to recover this mapping. The transformer is optimized with gradient flow (GF), spectral GF, or Sign GF, which are…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2610.02798
curl -X POST /api/v1/records/arxiv/2610.02798/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2610.02798/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'