PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$-Norm Steepest Descent

Yao Lu, Dengdong Fan, Shixun Zhang, Yonghong Tian

arXivarxiv/2605.10335May 11, 2026cs.LGscore 0

Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by $\ell_p$-norm steepest descent, PowerStep applies a signed-power transform directly to one momentum buffer. We establish a finite-horizon stationarity bound for exact, unregularized updates, with an $O(1/\sqrt{T})$ term and a noise-dependent residual. Experiments on Transformers from 124M to…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2605.10335
curl -X POST /api/v1/records/arxiv/2605.10335/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.10335/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'