Towards a Statistical Understanding of Mixture-of-Experts

Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang

arXivarxiv/2609.03501Sep 3, 2026stat.MLscore 0

Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small subset of experts for each input. Despite their growing importance in modern large-scale models, the statistical roles of their design choices, especially routing, sparse activation, and shared experts, remain only partially understood, as existing theory has largely focused on parametric or correctly specified MoE models. In this paper, we view MoE as a form of localized aggregation and show how this localization…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2609.03501
curl -X POST /api/v1/records/arxiv/2609.03501/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2609.03501/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'