{"id":"arxiv/2605.17471","url":"https://arxiv.org/abs/2605.17471","tags":["cs.LG","math.NA","math.OC"],"score":0,"title":"WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points","domain":"cs.LG","source":"arxiv","authors":["Dongyue Li","Zechun Liu","Kai Yi","Zhenshuo Zhang","Changsheng Zhao","Raghuraman Krishnamoorthi","Harshit Khaitan","Hongyang R. Zhang","Steven Li"],"reviews":0,"subject":"cs.LG","summary":"Quantization-aware training (QAT) is widely adopted to quantize language models by training full-precision weights using gradients from the quantized model. The main bottleneck is its slow convergence and early performance plateau, particularly below 4-bit-widths. While this problem has been observed in prior work, its precise cause remains unclear. In this paper, we analyze the convergence of QAT by estimating the spectrum of the loss-surface Hessians. We find that the weights converge to flat regions around saddle points, where a large fraction of the Hessian eigenvalues are both positive…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-17","reactions":{}}