{"id":"arxiv/2605.11316","url":"https://arxiv.org/abs/2605.11316","tags":["cs.LG","math.OC"],"score":0,"title":"Error whitening: Why Gauss-Newton outperforms Newton","domain":"cs.LG","source":"arxiv","authors":["Maricela Best McKay","Nathan P. Lawrence","Brian Wetton","R. Bhushan Gopaluni"],"reviews":0,"subject":"cs.LG","summary":"The Gauss-Newton matrix is widely viewed as a positive semidefinite approximation of the Hessian, yet mounting empirical evidence shows that Gauss-Newton descent outperforms Newton's method. We adopt a function space perspective to analyze this phenomenon. We show that the generalized Gauss-Newton (GGN) matrix projects the Newton direction in function space onto the model's tangent space, while a Jacobian-only variant obtained by applying the least squares Gauss-Newton matrix to non-least squares losses projects the function space loss gradient onto this same tangent space. Both projections…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-11","reactions":{}}