{"id":"arxiv/2605.08292","url":"https://arxiv.org/abs/2605.08292","tags":["cs.LG","cs.AI","math.OC"],"score":0,"title":"Hierarchical Mixture-of-Experts with Two-Stage Optimization","domain":"cs.LG","source":"arxiv","authors":["Gleb Molodtsov","Alexander Miasnikov","Aleksandr Beznosikov"],"reviews":0,"subject":"cs.LG","summary":"Sparse Mixture-of-Experts (MoE) models scale capacity by routing each token to a small subset of experts. However, their routers exhibit a fundamental trade-off: strong load balancing can suppress expert specialization, while aggressive diversity often causes routing collapse. We propose Hi-MoE, a grouped MoE framework that decomposes routing control into two coupled levels: (i) inter-group balancing that enforces fair traffic across expert groups, and (ii) intra-group specialization that promotes complementary expert behaviors while preventing within-group collapse. Our analysis provides a…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-08","reactions":{}}