$φ$-Balancing for Mixture-of-Experts Training

Lizhang Chen; Jonathan Li; Qi Wang; Runlong Liao; Shuozhe Li; Chen Liang; Ni Lao; Qiang Liu

$φ$-Balancing for Mixture-of-Experts Training

Lizhang Chen, Jonathan Li, Qi Wang, Runlong Liao, Shuozhe Li, Chen Liang, Ni Lao, Qiang Liu

arXivarxiv/2605.15403May 14, 2026cs.LGscore 0

Mixture-of-Experts (MoE) models rely on balanced expert utilization to fully realize their scalability. However, existing load-balancing methods are largely heuristic and operate on noisy mini-batch assignment statistics, introducing bias relative to population-level objectives. We propose $φ$-balancing, a principled framework that directly targets population-level expert balance by minimizing a strictly convex, symmetric, and differentiable potential of the expected routing distribution. Using convex duality, we derive an equivalent min-max formulation and obtain a simple online algorithm…

Source JSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2605.15403
curl -X POST /api/v1/records/arxiv/2605.15403/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.15403/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'