{"id":"arxiv/2605.03869","url":"https://arxiv.org/abs/2605.03869","tags":["cs.LG","math.OC"],"score":0,"title":"On Adaptivity in Zeroth-Order Optimization","domain":"cs.LG","source":"arxiv","authors":["Hassan Dbouk","Nidham Gazagnadou","Matthias Reisser","Christos Louizos"],"reviews":0,"subject":"cs.LG","summary":"We investigate the effectiveness of adaptive zeroth-order (ZO) optimization for memory-constrained fine-tuning of large language models (LLMs). Contrary to prior claims, we show that adaptive ZO methods such as ZO-Adam offer no convergence advantage over well-tuned ZO-SGD, while incurring significant memory overhead. Our analysis reveals that in high dimensions, ZO gradients lack coordinate-wise heterogeneity, rendering adaptive mechanisms memory inefficient. Leveraging this insight, we propose MEAZO, a memory-efficient adaptive ZO optimizer that tracks only a single scalar for global step…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-05","reactions":{}}