{"id":"arxiv/2606.23676","url":"https://arxiv.org/abs/2606.23676","tags":["cs.LG","cs.AI","math.OC","stat.ML"],"score":0,"title":"Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?","domain":"cs.LG","source":"arxiv","authors":["Dingzhi Yu","Hongyi Tao","Yuanyu Wan","Luo Luo","Lijun Zhang"],"reviews":0,"subject":"cs.LG","summary":"AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying, as empirical evidence indicates that stochastic gradient noise in LLM pretraining is typically heavy-tailed. Recent work shows that sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise. However, no rigorous convergence theory for AdamW has yet been established in this regime. Can AdamW converge under the same heavy-tailed…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-06-22","reactions":{}}