Cache-to-Cache: Direct Semantic Communication Between Large Language Models
Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, Yu Wang
TL;DR
This work addresses the inefficiency and information loss inherent in text-based inter-LLM communication by proposing Cache-to-Cache (C2C), a paradigm that directly fuses KV-Cache representations across models. Through oracle studies and a dedicated fuser architecture, C2C demonstrates that cross-model KV-Cache transformation and selective layer-wise fusion can yield meaningful accuracy gains (8.5–10.5% on average) and substantial latency reductions (around 2×) compared to text-based communication. The approach includes token and layer alignment, a gated, residual fusion mechanism, and a staged training scheme that preserves the integrity of both Sharer and Receiver while learning to leverage shared semantic information. Empirically, C2C scales with longer contexts and stronger Sharers, generalizes across model families and sizes, and remains robust against ablations, indicating practical potential for scalable, low-latency multi-LLM systems. The findings suggest that internal KV-Cache semantics can serve as a rich, low-latency medium for cross-model collaboration beyond natural language prompts, with wide applicability in privacy-preserving, edge-cloud setups and multimodal integrations.
Abstract
Multi-LLM systems harness the complementary strengths of diverse Large Language Models, achieving performance and efficiency gains unattainable by a single model. In existing designs, LLMs communicate through text, forcing internal representations to be transformed into output token sequences. This process both loses rich semantic information and incurs token-by-token generation latency. Motivated by these limitations, we ask: Can LLMs communicate beyond text? Oracle experiments show that enriching the KV-Cache semantics can improve response quality without increasing cache size, supporting KV-Cache as an effective medium for inter-model communication. Thus, we propose Cache-to-Cache (C2C), a new paradigm for direct semantic communication between LLMs. C2C uses a neural network to project and fuse the source model's KV-cache with that of the target model to enable direct semantic transfer. A learnable gating mechanism selects the target layers that benefit from cache communication. Compared with text communication, C2C utilizes the deep, specialized semantics from both models, while avoiding explicit intermediate text generation. Experiments show that C2C achieves 8.5-10.5% higher average accuracy than individual models. It further outperforms the text communication paradigm by approximately 3.0-5.0%, while delivering an average 2.0x speedup in latency. Our code is available at https://github.com/thu-nics/C2C.