Next Token Perception Score: Analytical Assessment of your LLM Perception Skills

TL;DR

This work tackles the inconsistent transfer of autoregressive representations to downstream perception tasks by introducing the Next Token Perception Score (NTPS), a closed-form metric that quantifies subspace alignment between perception and autoregressive features. In a linear regime, NTPS = (||P U||^2_F)/(||U||^2_F) with P = V^T V, and it is shown to bound the excess regression loss incurred by autoregressive encoders relative to an optimal perceptual encoder. Empirically, NTPS correlates strongly with linear-probing accuracy across 12 NLP tasks and 8 pretrained models (ranging from 270M to 8B parameters) and reliably increases after LoRA finetuning, suggesting improved subspace overlap. Importantly, NTPS can predict the magnitude of LoRA gains, enabling a lightweight prescreening tool for adaptation and guiding decisions about parameter-efficient fine-tuning. Overall, NTPS provides both theoretical insight and practical utility for analytically assessing and improving LLM perception skills.

Abstract

Autoregressive pretraining has become the de facto paradigm for learning general-purpose representations in large language models (LLMs). However, linear probe performance across downstream perception tasks shows substantial variability, suggesting that features optimized for next-token prediction do not consistently transfer well to downstream perception tasks. We demonstrate that representations learned via autoregression capture features that may lie outside the subspaces most informative for perception. To quantify the (mis)alignment between autoregressive pretraining and downstream perception, we introduce the Next Token Perception Score (NTPS)-a score derived under a linear setting that measures the overlap between autoregressive and perception feature subspaces. This metric can be easily computed in closed form from pretrained representations and labeled data, and is proven to both upper- and lower-bound the excess loss. Empirically, we show that NTPS correlates strongly with linear probe accuracy across 12 diverse NLP datasets and eight pretrained models ranging from 270M to 8B parameters, confirming its utility as a measure of alignment. Furthermore, we show that NTPS increases following low-rank adaptation (LoRA) fine-tuning, especially in large models, suggesting that LoRA aligning representations to perception tasks enhances subspace overlap and thus improves downstream performance. More importantly, we find that NTPS reliably predicts the additional accuracy gains attained by LoRA finetuning thereby providing a lightweight prescreening tool for LoRA adaptation. Our results offer both theoretical insights and practical tools for analytically assessing LLM perception skills.

Figures (3)

• Figure 1: Correlation between NTPS and downstream MSE loss (rows 1 and 2), and between NTPS and accuracy (rows 3 and 4), with dashed lines indicating linear regression fits. Higher alignment scores correspond to lower MSE loss and higher accuracy in downstream tasks.
• Figure 2: Correlation between NTPS and accuracy gain (LoRA finetuning accuracy-linear probing accuracy), with dashed lines indicating linear regression fits. Higher alignment scores correspond to lower accuracy gain after LoRA finetuning in downstream tasks.
• Figure : Computation of NTPS

Theorems & Definitions (9)

• Lemma 1: Equivalence between MSE and CE
• Theorem 1
• Theorem 2: Excess regression loss vs. subspace alignment
• Lemma 1: Equivalence between MSE and CE
• proof
• Theorem 2
• proof
• Theorem 2: Excess regression loss vs. subspace alignment
• proof