{"id":"arxiv/2609.36945","url":"https://arxiv.org/abs/2609.36945","tags":["cs.LG","math.OC"],"score":0,"title":"Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness","domain":"cs.LG","source":"arxiv","authors":["Zhiwei Wang","Yanxi Chen","Yaliang Li","Bolin Ding"],"reviews":0,"subject":"cs.LG","summary":"We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version of REINFORCE -- referred to as RE(S) -- that updates the rollout distribution once every $S \\ge 1$ gradient steps. Prior work in bandits and reinforcement learning has developed rich theory for policy gradient methods, and on-policy sampling (i.e., a small $S$, ideally $1$) is often viewed as crucial to their success; yet in prominent application like post-training large language models, reward-guided self-training has proved to be effective…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-09-29","reactions":{}}