{"id":"arxiv/2609.18314","url":"https://arxiv.org/abs/2609.18314","tags":["cs.LG","math.OC","stat.ML"],"score":0,"title":"Beyond Quadratic Loss: The Stability Phase Diagram of Adam","domain":"cs.LG","source":"arxiv","authors":["Gaoxiang Tang","Huanran Chen","Ziming Liu"],"reviews":0,"subject":"cs.LG","summary":"Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the $(β_1,β_2)$ plane. Across a range of model--task settings, an approximately linear boundary, $1-β_2=C(1-β_1)$, separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-09-16","reactions":{}}