{"id":"arxiv/2607.05872","url":"https://arxiv.org/abs/2607.05872","tags":["cs.LG","math.OC","stat.ML"],"score":0,"title":"No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training","domain":"cs.LG","source":"arxiv","authors":["Noel Thomas"],"reviews":0,"subject":"cs.LG","summary":"Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked. We show that beyond a small reproducible core, there is no such object. Two estimates of the top-r subspace computed at the same step from disjoint minibatches disagree as much as estimates computed T steps apart (0.73 vs 0.74 of the maximal chordal distance sqrt(2r), at Pythia-160M with r=128): the apparent rotation at each refresh is dominated by estimator noise. This holds…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-07-07","reactions":{}}