{"id":"arxiv/2610.39412","url":"https://arxiv.org/abs/2610.39412","tags":["cs.LG","cs.DC","math.OC"],"score":0,"title":"Dobb: Shard-Local Preconditioning in Block-Diagonal Eigenbases","domain":"cs.LG","source":"arxiv","authors":["Shi-Yuk Fong","Shuhua Yu","Jiawei Zhang"],"reviews":0,"subject":"cs.LG","summary":"Training language models whose parameters and optimizer state exceed a single node's memory requires sharding these tensors across devices. Under tensor parallelism (TP) or fully sharded data parallelism (FSDP), matrix-aware optimizers such as Muon introduce communication to perform global matrix operations across shards. We propose Dobb (Distributed Optimizer with Block-diagonal Eigenbases), a matrix-aware optimizer designed to compute updates locally within parameter shards. Dobb approximates the input-activation second moment with diagonal blocks and tracks a complete basis within each…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-10-29","reactions":{}}