LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States

TL;DR

This work questions the traditional reliance on last-layer hidden states for LLM-based sentence embeddings and proposes that value vectors carry richer sentence-level semantics. It introduces Value Aggregation (VA), a training-free method that pools value vectors across selected layers and tokens, and extends this with Aligned Weighted VA (AlignedWVA) by projecting weighted values into the residual-stream space, achieving state-of-the-art performance among training-free embeddings. The authors support their claims with extensive experiments on LLaMA-2 and Qwen-3 backbones, showing VA often outperforms prompt-free baselines and rivals costlier ensemble approaches like MetaEOL, while offering substantial efficiency gains. They further demonstrate that finetuning VA with lightweight LoRA-based strategies can approach or match fine-tuned hidden-state methods using far fewer trainable parameters, highlighting the promise of value-space representations for scalable and effective sentence embeddings.

Abstract

Sentence representations are foundational to many Natural Language Processing (NLP) applications. While recent methods leverage Large Language Models (LLMs) to derive sentence representations, most rely on final-layer hidden states, which are optimized for next-token prediction and thus often fail to capture global, sentence-level semantics. This paper introduces a novel perspective, demonstrating that attention value vectors capture sentence semantics more effectively than hidden states. We propose Value Aggregation (VA), a simple method that pools token values across multiple layers and token indices. In a training-free setting, VA outperforms other LLM-based embeddings, even matches or surpasses the ensemble-based MetaEOL. Furthermore, we demonstrate that when paired with suitable prompts, the layer attention outputs can be interpreted as aligned weighted value vectors. Specifically, the attention scores of the last token function as the weights, while the output projection matrix ($W_O$) aligns these weighted value vectors with the common space of the LLM residual stream. This refined method, termed Aligned Weighted VA (AlignedWVA), achieves state-of-the-art performance among training-free LLM-based embeddings, outperforming the high-cost MetaEOL by a substantial margin. Finally, we highlight the potential of obtaining strong LLM embedding models through fine-tuning Value Aggregation.

Figures (9)

• Figure 1: The value aggregation architecture, which involves pooling the token-level value representations across all layers.
• Figure 2: Layer-wise segment matching on LongBench v2. We split each long sequence into a prefix and suffix segment (split point sampled between $1/4$ and $3/4$ of the token length), and retrieve the matching suffix segment for each prefix segment using embeddings from each layer. Each panel reports recall@k (left: $k{=}1$, middle: $k{=}5$, right: $k{=}10$). VA (of l-layer) improves steadily with depth and outperforms HS (of l-layer) in deeper layers across all $k$, while the hidden state is competitive only in very early layers.
• Figure 3: Score by Layers for Llama (Left) and Qwen (Right)
• Figure 4: Results on the "Average of Major Task Categories" with VA and other finetuned models (LLama-2 (7B) backbone)
• Figure 5: Architecture for Probing Next Token Prediction.

Theorems & Definitions (6)

• Definition 2.1: Sentence-Level Semantics
• Definition 2.2: Probabilistic Truth Function
• Definition 4.1: Autoregressive next-token prediction and token-level NLL
• Definition 4.2: InfoNCE loss
• Proposition 4.3: Token-level NLL is an InfoNCE instance
• proof