{"id":"arxiv/2605.10671","url":"https://arxiv.org/abs/2605.10671","tags":["cs.LG","math.OC","stat.ML"],"score":0,"title":"Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework","domain":"cs.LG","source":"arxiv","authors":["Phalguni Nanda","Zaiwei Chen"],"reviews":0,"subject":"cs.LG","summary":"In this work, we show that natural policy gradient, a core algorithm in reinforcement learning, admits an exact formulation as a smoothed and averaged form of policy iteration. Specifically, we introduce doubly smoothed policy iteration (DSPI), a Bellman-operator framework in which each policy is obtained by applying a regularized greedy step to a weighted average of past $Q$-functions. DSPI includes policy iteration, dual-averaged policy iteration, natural policy gradient, and more general policy dual averaging methods as special cases. Using only monotonicity and contraction of smoothed…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-11","reactions":{}}