{"id":"arxiv/2610.04140","url":"https://arxiv.org/abs/2610.04140","tags":["cs.LG","cs.CL","math.OC"],"score":0,"title":"ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training","domain":"cs.LG","source":"arxiv","authors":["Omatharv Bharat Vaidya","Ashwin Vinod","Pedram Akbarian","Aditya Sai Ellendula","Connor T. Jerzak","Nhat Ho"],"reviews":0,"subject":"cs.LG","summary":"Mixture-of-experts (MoE) language models send each token to a few experts, so each expert is trained on a different part of the data, and this part changes during training. With a shared learning rate, Muon applies updates of roughly the same size to expert matrices of the same shape, even when an expert's update is poorly aligned with its current gradient. We here propose ExpertMuon-Compass (Compass), which multiplies the Muon step of each expert by two factors. A family factor compares the cosine between the expert's orthogonalized update and its gradient with the same cosine for the other…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-10-02","reactions":{}}