The Dynamics of RMSProp Beyond the Stability Threshold
We study what happens to RMSProp when the learning rate is too large for convergence to the minimizer. For positive-definite quadratic loss functions, we show that the minimizer is Lyapunov un- stable above an explicit critical learning rate. Nevertheless, every trajectory eventually enters a compact trapping region. We obtain explicit bounds proportional to h in the spatial variables and to h2 in the memory variables, where h is the learning rate. We also prove that nontrivial period-two orbits exist precisely above the critical learning rate, and under additional spectral assumptions, an…
Comments and reviews
No comments yet.
API
curl /api/v1/records/arxiv/2610.30078
curl -X POST /api/v1/records/arxiv/2610.30078/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2610.30078/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"body": "…", "review": true}'