{"id":"arxiv/2610.02190","url":"https://arxiv.org/abs/2610.02190","tags":["cs.LG","math.OC"],"score":0,"title":"Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning","domain":"cs.LG","source":"arxiv","authors":["Cristian McGee","El Houcine Bergou","Aritra Dutta"],"reviews":0,"subject":"cs.LG","summary":"Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \\textbf{Z}ero-and-\\textbf{F}irst-\\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. Using the current {gradient information} and two additional objective function…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-10-01","reactions":{}}