Perturbation is All You Need for Extrapolating Language Models

Zetai Cen, Jin Zhu, Xinwei Shen, Chengchun Shi

arXivarxiv/2605.04344May 5, 2026stat.MLscore 0

This paper develops a statistical theory of extrapolation for large language models, by reinterpreting them through pre-post-additive noise models. In contrast to the standard autoregressive next-token prediction based on an exact prefix, we introduce a perturbation-based procedure that first transforms the prefix into a semantic neighbour and then conditions on this perturbed variant for next-token prediction. This yields a hierarchical model with a pre-post-additive noise structure. Within this framework, we develop a rigorous theory of extrapolability, namely, the capacity of a model…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2605.04344
curl -X POST /api/v1/records/arxiv/2605.04344/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.04344/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'