{"id":"arxiv/2605.10335","url":"https://arxiv.org/abs/2605.10335","tags":["cs.LG","cs.AI","cs.CL","math.NA","math.OC"],"score":0,"title":"PowerStep: Memory-Efficient Adaptive Optimization via $\\ell_p$-Norm Steepest Descent","domain":"cs.LG","source":"arxiv","authors":["Yao Lu","Dengdong Fan","Shixun Zhang","Yonghong Tian"],"reviews":0,"subject":"cs.LG","summary":"Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by $\\ell_p$-norm steepest descent, PowerStep applies a signed-power transform directly to one momentum buffer. We establish a finite-horizon stationarity bound for exact, unregularized updates, with an $O(1/\\sqrt{T})$ term and a noise-dependent residual. Experiments on Transformers from 124M to…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-11","reactions":{}}