{"id":"arxiv/2601.16399","url":"https://arxiv.org/abs/2601.16399","tags":["cs.LG","math.OC"],"score":0,"title":"A Hessian-Free Actor-Critic Algorithm for Bi-Level Reinforcement Learning with Applications to LLM Fine-Tuning","domain":"cs.LG","source":"arxiv","authors":["Sihan Zeng","Sujay Bhatt","Sumitra Ganesh","Alec Koppel"],"reviews":0,"subject":"cs.LG","summary":"We study a structured bi-level optimization problem where the upper-level objective is a smooth function and the lower-level problem is policy optimization in a Markov decision process (MDP). The upper-level decision variable parameterizes the reward of the lower-level MDP, and the upper-level objective depends on the optimal induced policy. Existing methods for bi-level optimization and RL often require second-order information, impose strong regularization at the lower level, or inefficiently use samples through nested-loop procedures. In this work, we propose a single-loop, first-order…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-01-23","reactions":{}}