{"id":"arxiv/2602.05298","url":"https://arxiv.org/abs/2602.05298","tags":["stat.ML","cs.LG","math.OC"],"score":0,"title":"Logarithmic-time Schedules for Scaling Language Models with Momentum","domain":"stat.ML","source":"arxiv","authors":["Damien Ferbach","Courtney Paquette","Gauthier Gidel","Katie Everett","Elliot Paquette"],"reviews":0,"subject":"stat.ML","summary":"In practice, the hyperparameters $(β_1, β_2)$ and weight-decay $λ$ in AdamW are typically kept at fixed values. Is there any reason to do otherwise? We show that for large-scale language model training, the answer is yes: by exploiting the power-law structure of language data, one can design time-varying schedules for $(β_1, β_2, λ)$ that deliver substantial performance gains. We study logarithmic-time scheduling, in which the optimizer's gradient memory horizon grows with training time. Although naive variants of this are unstable, we show that suitable damping mechanisms restore stability…","category":"research","comments":0,"disputed":false,"division":"stat","mentions":0,"discussed":0,"published":"2026-02-05","reactions":{}}