Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

You Need an Encoder for Native Position-Independent Caching

Shiju Zhao, Junhao Hu, Jiaqi Zheng, Guihai Chen

TL;DR

This work tackles the inefficiency of prefix-based KV caching in LLMs when contexts arrive in arbitrary orders. It introduces native Position-Independent Caching by reintroducing a trainable encoder into decoder-only LLMs and training it for PIC, forming a PIC-aware architecture named Comb. Coupled with a PIC management system (COMB) that integrates with existing inference stacks, Comb achieves up to $TTFT$ reductions of up to $94\%$ and up to $3\times$ throughput gains while maintaining or surpassing the accuracy of prefix caching, and it reduces KV memory by about 75\%. The approach is demonstrated across Llama-3.1-8B-Instruct and DeepSeek-V2-Lite-Chat, showing robustness and non-intrusiveness, and highlighting PIC’s potential for efficient retrieval in AI agents and RAG workflows.

Abstract

The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Caching (PIC) has been proposed to enable KV reuse without positional constraints; however, existing approaches often incur substantial accuracy degradation, limiting their practical adoption. To address this issue, we propose native PIC by reintroducing the encoder to prevalent decoder-only LLMs and explicitly training it to support PIC. We further develop COMB, a PIC-aware caching system that integrates seamlessly with existing inference frameworks. Experimental results show that COMB reduces Time-to-First-Token (TTFT) by 51-94% and increases throughput by 3$\times$ with comparable accuracy. Furthermore, the quality improvement when using DeepSeek-V2-Lite-Chat demonstrates the applicability of COMB to other types of decoder-only LLMs. Our code is available at https://github.com/shijuzhao/Comb.

You Need an Encoder for Native Position-Independent Caching

TL;DR

This work tackles the inefficiency of prefix-based KV caching in LLMs when contexts arrive in arbitrary orders. It introduces native Position-Independent Caching by reintroducing a trainable encoder into decoder-only LLMs and training it for PIC, forming a PIC-aware architecture named Comb. Coupled with a PIC management system (COMB) that integrates with existing inference stacks, Comb achieves up to reductions of up to and up to throughput gains while maintaining or surpassing the accuracy of prefix caching, and it reduces KV memory by about 75\%. The approach is demonstrated across Llama-3.1-8B-Instruct and DeepSeek-V2-Lite-Chat, showing robustness and non-intrusiveness, and highlighting PIC’s potential for efficient retrieval in AI agents and RAG workflows.

Abstract

The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Caching (PIC) has been proposed to enable KV reuse without positional constraints; however, existing approaches often incur substantial accuracy degradation, limiting their practical adoption. To address this issue, we propose native PIC by reintroducing the encoder to prevalent decoder-only LLMs and explicitly training it to support PIC. We further develop COMB, a PIC-aware caching system that integrates seamlessly with existing inference frameworks. Experimental results show that COMB reduces Time-to-First-Token (TTFT) by 51-94% and increases throughput by 3 with comparable accuracy. Furthermore, the quality improvement when using DeepSeek-V2-Lite-Chat demonstrates the applicability of COMB to other types of decoder-only LLMs. Our code is available at https://github.com/shijuzhao/Comb.
Paper Structure (29 sections, 13 equations, 10 figures, 1 table)

This paper contains 29 sections, 13 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Autoregressive Generation and KV Cache
  4. Position-Independent Caching (PIC)
  5. System Overview
  6. Model Design
  7. Self-Attention Layer
  8. Cross-Attention Layer
  9. Complexity Analysis
  10. Time Complexity.
  11. Memory Complexity.
  12. Implementation
  13. Models
  14. Training Data and Recipe
  15. System Details
  16. ...and 14 more sections

Figures (10)

  • Figure 1: Position-Independent Caching (PIC) workflow. $KV_i$, $KV_j$, $KV_k$ indicate KV vectors in arbitrary order/position.
  • Figure 2: Comb Overview.
  • Figure 3: Model architecture.
  • Figure 4: Prefill and decode length distribution.
  • Figure 5: Comparison of TTFT ($\downarrow$ Better) and accuracy ($\uparrow$ Better) using Llama-3.1-8B-Instruct. Note that the TTFT of BlockAttention is not shown because it is not integrated with vLLM currently.
  • ...and 5 more figures