Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning

Zhaiming Shen, Alexander Cloninger, Rongjie Lai, Wenjing Liao

arXivarxiv/2610.24441Oct 17, 2026cs.LGscore 0

Transformers have emerged as a dominant architecture in modern machine learning, with in-context learning (ICL) enabling models to adapt to new tasks by conditioning on prompts, without any parameter updates. To understand the attention mechanism in transformer blocks, a growing line of theoretical work interprets ICL through the lens of classical statistical estimators. Recent works have established a connection between the attention mechanism and kernel methods, while the kernel function is usually assumed to be fixed. In practice, the query and key matrices in the attention mechanism are…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2610.24441
curl -X POST /api/v1/records/arxiv/2610.24441/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2610.24441/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'