{"id":"arxiv/2609.07666","url":"https://arxiv.org/abs/2609.07666","tags":["cs.LG","math.OC"],"score":0,"title":"MpSub: A Momentum $p$-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models","domain":"cs.LG","source":"arxiv","authors":["Yuyang Wang","Haoyu Yao","Pengcheng Xie"],"reviews":0,"subject":"cs.LG","summary":"Full-parameter fine-tuning of large language models has substantial memory costs because backpropagation stores activations and gradients. Zeroth-order optimization avoids this by estimating update directions from loss evaluations, but existing methods require tuning a sensitive learning rate for each model and task. We propose the momentum $p$-dimensional subspace trust-region method (MpSub). At each iteration, MpSub searches within a $p$-dimensional subspace: one direction preserves historical momentum from the most recent accepted step, while the remaining directions explore via fresh…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-09-07","reactions":{}}