{"id":"arxiv/2608.16760","url":"https://arxiv.org/abs/2608.16760","tags":["cs.LG","math.OC"],"score":0,"title":"On the Principles Behind Neural Network Optimizers","domain":"cs.LG","source":"arxiv","authors":["Yushun Zhang"],"reviews":0,"subject":"cs.LG","summary":"Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation. This thesis develops a principled grounding for Adam and motivates new designs. First, we revisit Adam's divergence--convergence debate and show the existence of a problem-dependent phase transition: with properly chosen, batch-size-dependent hyperparameters, Adam converges, whereas under small-$β_2$ regimes it can diverge. Second, we investigate why Adam substantially outperforms SGD on Transformers through Hessian structure. We find that the…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-08-17","reactions":{}}