Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs

Yixuan Tang, Yi Yang

TL;DR

KV-Embedding tackles the problem of obtaining high-quality text embeddings from frozen decoder-only LLMs without fine-tuning. It leverages internal KV states by re-routing the final token’s KV pair as a global prefix in selected layers, enabling all tokens to access a sequence-wide summary in a single forward pass. An intrinsic dimensionality–based layer selection scheme identifies where semantic compression is maximal, ensuring model-agnostic applicability. Across MTEB and LoCoV1 benchmarks on Qwen, Mistral, and Llama backbones, KV-Embedding consistently outperforms training-free baselines by up to 10% and maintains robustness up to 4{,}096 tokens, while improving embedding space isotropy and semantic alignment. This work highlights the potential of internal state manipulation as an efficient alternative to input modification for representation learning in large language models.

Abstract

While LLMs are powerful embedding backbones, their application in training-free settings faces two structural challenges: causal attention restricts early tokens from accessing subsequent context, and the next-token prediction objective biases representations toward generation rather than semantic compression. To address these limitations, we propose KV-Embedding, a framework that activates the latent representation power of frozen LLMs. Our method leverages the observation that the key-value (KV) states of the final token at each layer encode a compressed view of the sequence. By re-routing these states as a prepended prefix, we enable all tokens to access sequence-level context within a single forward pass. To ensure model-agnostic applicability, we introduce an automated layer selection strategy based on intrinsic dimensionality. Evaluations on MTEB across Qwen, Mistral, and Llama backbones show that KV-Embedding outperforms existing training-free baselines by up to 10%, while maintaining robust performance on sequences up to 4,096 tokens. These results demonstrate that internal state manipulation offers an efficient alternative to input modification, and we hope this work encourages further exploration of LLM internals for representation learning.

KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs

TL;DR

KV-Embedding tackles the problem of obtaining high-quality text embeddings from frozen decoder-only LLMs without fine-tuning. It leverages internal KV states by re-routing the final token’s KV pair as a global prefix in selected layers, enabling all tokens to access a sequence-wide summary in a single forward pass. An intrinsic dimensionality–based layer selection scheme identifies where semantic compression is maximal, ensuring model-agnostic applicability. Across MTEB and LoCoV1 benchmarks on Qwen, Mistral, and Llama backbones, KV-Embedding consistently outperforms training-free baselines by up to 10% and maintains robustness up to 4{,}096 tokens, while improving embedding space isotropy and semantic alignment. This work highlights the potential of internal state manipulation as an efficient alternative to input modification for representation learning in large language models.

Abstract

While LLMs are powerful embedding backbones, their application in training-free settings faces two structural challenges: causal attention restricts early tokens from accessing subsequent context, and the next-token prediction objective biases representations toward generation rather than semantic compression. To address these limitations, we propose KV-Embedding, a framework that activates the latent representation power of frozen LLMs. Our method leverages the observation that the key-value (KV) states of the final token at each layer encode a compressed view of the sequence. By re-routing these states as a prepended prefix, we enable all tokens to access sequence-level context within a single forward pass. To ensure model-agnostic applicability, we introduce an automated layer selection strategy based on intrinsic dimensionality. Evaluations on MTEB across Qwen, Mistral, and Llama backbones show that KV-Embedding outperforms existing training-free baselines by up to 10%, while maintaining robust performance on sequences up to 4,096 tokens. These results demonstrate that internal state manipulation offers an efficient alternative to input modification, and we hope this work encourages further exploration of LLM internals for representation learning.
Paper Structure (58 sections, 8 equations, 3 figures, 21 tables, 1 algorithm)

This paper contains 58 sections, 8 equations, 3 figures, 21 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based Text Embeddings
  4. Training-Free LLM Embeddings
  5. KV Manipulation in Attention
  6. Method
  7. Preliminaries
  8. KV-Embedding
  9. Compression-Oriented Prompting
  10. Key-Value Re-Routing
  11. Automated Layer Selection via Intrinsic Dimensionality
  12. Experiments
  13. Experimental Setup
  14. Benchmarks.
  15. Models.
  16. ...and 43 more sections

Figures (3)

  • Figure 1: Overview of KV-Embedding. (a) Standard causal attention creates information asymmetry, as tokens only attend to preceding context. (b) KV-Embedding re-routes the last KV pair as a global prefix, enabling sequence-wide context access within one forward pass. (c) The pipeline identifies optimal re-routing anchors by locating layers with minimum intrinsic dimensionality, ensuring model-agnostic stability without manual tuning.
  • Figure 2: Average performance of Mistral-7B-Instruct with KV-Embedding across different attention bias values. A bias value of 1.0 corresponds to the highest average score.
  • Figure 3: Intrinsic Dimensionality (ID) across layers for Mistral-7B-Instruct-v0.1 and Qwen3-4B. Different architectures exhibit distinct semantic compression patterns.