{"id":"arxiv/2610.02355","url":"https://arxiv.org/abs/2610.02355","tags":["cs.LG","math.OC"],"score":0,"title":"Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance","domain":"cs.LG","source":"arxiv","authors":["Arda Fazla","Antesh Upadhyay","Ege C. Kaya","M. Berk Sahin","Abolfazl Hashemi"],"reviews":0,"subject":"cs.LG","summary":"Increasing the batch size during training is a common practice in large language model (LLM) pretraining, yet the theoretical justification behind its success is not well understood. Analyses of stochastic optimization often assume uniformly bounded stochastic gradient variance, yet recent evidence suggests that this assumption fails in many practical nonconvex problems. The Blum--Gladyshev (BG-$0$) noise model relaxes this assumption by allowing the variance to grow quadratically with the distance from initialization, suggesting that batch size schedulers can help by controlling the…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-10-01","reactions":{}}