{"id":"arxiv/2607.23777","url":"https://arxiv.org/abs/2607.23777","tags":["cs.LG","cond-mat.dis-nn","cs.AI","math.OC"],"score":0,"title":"Scale Weight Decay and Train Better","domain":"cs.LG","source":"arxiv","authors":["Anuj Apte"],"reviews":0,"subject":"cs.LG","summary":"The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data. This is typically done with a constant decoupled weight decay which causes the network weights to shrink steadily over the course of training. Taking inspiration from the Robbins--Monro conditions, we propose to scale weight decay by the fraction of the peak learning rate $η/η_{\\max}$. We prove that this scaled weight decay preserves the asymptotic stationarity guarantees of the corresponding unregularized methods for both stochastic gradient descent and the non-Euclidean spectral…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-07-26","reactions":{}}