{"id":"arxiv/2605.11838","url":"https://arxiv.org/abs/2605.11838","tags":["cs.LG","math.OC"],"score":0,"title":"Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters","domain":"cs.LG","source":"arxiv","authors":["Alexander Yukhimchuk","Mladen Kolar","Martin Takáč","Sayantan Choudhury"],"reviews":0,"subject":"cs.LG","summary":"Gradient clipping is a standard safeguard for training neural networks under noisy, heavy-tailed stochastic gradients; yet, most clipping rules treat all parameters as vectors and ignore the matrix structure of modern architectures. We show empirically that data outliers often amplify only a small number of leading singular values in layer-wise gradient matrices, while the rest of the spectrum remains largely unchanged. Motivated by this phenomenon, we propose spectral clipping, which stabilizes training by clamping singular values that exceed a threshold while preserving the singular…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-12","reactions":{}}