Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

Zihan Zhu; Zhehang Du; Xuyang Chen; Tim Tsz-Kit Lau; Jiayuan Wu; X. Y. Han; Qi Long; Weijie Su

Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

Zihan Zhu, Zhehang Du, Xuyang Chen, Tim Tsz-Kit Lau, Jiayuan Wu, X. Y. Han, Qi Long, Weijie Su

arXivarxiv/2609.12123Sep 10, 2026cs.LGscore 0

Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representational capacity, the optimizer shapes how much of that capacity is used in the induced weight-space updates. In a case study of GPT-2 adaptation with LoRA, we observe a strong rank-dependent optimizer effect. Despite using the same nominal rank, AdamW often produces per-step…

Source JSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2609.12123
curl -X POST /api/v1/records/arxiv/2609.12123/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2609.12123/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'