{"id":"arxiv/2609.30957","url":"https://arxiv.org/abs/2609.30957","tags":["cs.LG","math.OC"],"score":0,"title":"Towards Understanding Momentum Acceleration in River-Valley Loss Landscape","domain":"cs.LG","source":"arxiv","authors":["Miao Lu","Zeyu Bian","Kaiyue Wen","Beining Wu","Siyu Chen","Tianhao Wang","Zhiyuan Li"],"reviews":0,"subject":"cs.LG","summary":"The empirical success of pretraining large language models has inspired a deeper investigation into the underlying loss landscapes and the optimization dynamics. Recent empirical and theoretical study suggest that the training loss landscape often exhibits a \"river-valley\" structure, which features a low-loss manifold (river) flanked by sharp orthogonal directions with higher loss (mountains). In the long term, the optimization progress is determined primarily by the progress along the river. Within such a landscape, gradient descent with large learning rates can move faster along the river…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-09-25","reactions":{}}