{"id":"arxiv/2608.19491","url":"https://arxiv.org/abs/2608.19491","tags":["cs.LG","cs.CL","math.OC","stat.ML"],"score":0,"title":"Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule","domain":"cs.LG","source":"arxiv","authors":["Euijin Hong","Guannan Qu"],"reviews":0,"subject":"cs.LG","summary":"Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate. However, the inputs a deep network sees during training can be highly anisotropic, with a few directions queried frequently while most are seen rarely. Preconditioning methods address this anisotropy by wrapping extra processing around this buffer and leave the momentum update itself unchanged. We propose Activation-Keyed Momentum (AK-Momentum), which builds direction-awareness into the momentum update rule. The gradient of a linear layer splits…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-08-19","reactions":{}}