Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Modal Memory Compression for Efficient Multi-Agent Debate

Jing Wu, Yue Sun, Tianpei Xie, Suiyao Chen, Jingyuan Bao, Yaopengxiao Xu, Gaoyuan Du, Inseok Heo, Alexander Gutfraind, Xin Wang

TL;DR

This work tackles the token-growth problem in multi-agent debate (MAD) by introducing DebateOCR, a cross-modal framework that renders long textual histories as fixed-size images processed by a vision encoder and an adapter to generate vision tokens. The two-phase approach first trains a lightweight adapter to map SAM-CLIP features into the target LLM embedding space, then performs online rendering of debate histories into 1024×1024 images whose features are compressed into a fixed number of vision tokens, reducing total tokens from $O(K^2 R^2 L)$ to $O(KRN)$. Theoretical analysis based on the Information Bottleneck shows that, with diverse agents and sufficient per-agent information preservation, compressed histories concentrate near the bottleneck while artifact information is suppressed, enabling robust decisions from aggregated views. Empirically, DebateOCR achieves token reductions over 92% and meaningful speedups while maintaining or improving accuracy on GSM8K, MATH, and GPQA across multiple vision-language backbones, demonstrating scalability and practical impact for cross-modal MAD systems.

Abstract

Multi-agent debate can improve reasoning quality and reduce hallucinations, but it incurs rapidly growing context as debate rounds and agent count increase. Retaining full textual histories leads to token usage that can exceed context limits and often requires repeated summarization, adding overhead and compounding information loss. We introduce DebateOCR, a cross-modal compression framework that replaces long textual debate traces with compact image representations, which are then consumed through a dedicated vision encoder to condition subsequent rounds. This design compresses histories that commonly span tens to hundreds of thousands of tokens, cutting input tokens by more than 92% and yielding substantially lower compute cost and faster inference across multiple benchmarks. We further provide a theoretical perspective showing that diversity across agents supports recovery of omitted information: although any single compressed history may discard details, aggregating multiple agents' compressed views allows the collective representation to approach the information bottleneck with exponentially high probability.

Cross-Modal Memory Compression for Efficient Multi-Agent Debate

TL;DR

This work tackles the token-growth problem in multi-agent debate (MAD) by introducing DebateOCR, a cross-modal framework that renders long textual histories as fixed-size images processed by a vision encoder and an adapter to generate vision tokens. The two-phase approach first trains a lightweight adapter to map SAM-CLIP features into the target LLM embedding space, then performs online rendering of debate histories into 1024×1024 images whose features are compressed into a fixed number of vision tokens, reducing total tokens from to . Theoretical analysis based on the Information Bottleneck shows that, with diverse agents and sufficient per-agent information preservation, compressed histories concentrate near the bottleneck while artifact information is suppressed, enabling robust decisions from aggregated views. Empirically, DebateOCR achieves token reductions over 92% and meaningful speedups while maintaining or improving accuracy on GSM8K, MATH, and GPQA across multiple vision-language backbones, demonstrating scalability and practical impact for cross-modal MAD systems.

Abstract

Multi-agent debate can improve reasoning quality and reduce hallucinations, but it incurs rapidly growing context as debate rounds and agent count increase. Retaining full textual histories leads to token usage that can exceed context limits and often requires repeated summarization, adding overhead and compounding information loss. We introduce DebateOCR, a cross-modal compression framework that replaces long textual debate traces with compact image representations, which are then consumed through a dedicated vision encoder to condition subsequent rounds. This design compresses histories that commonly span tens to hundreds of thousands of tokens, cutting input tokens by more than 92% and yielding substantially lower compute cost and faster inference across multiple benchmarks. We further provide a theoretical perspective showing that diversity across agents supports recovery of omitted information: although any single compressed history may discard details, aggregating multiple agents' compressed views allows the collective representation to approach the information bottleneck with exponentially high probability.
Paper Structure (51 sections, 3 theorems, 57 equations, 3 figures, 3 tables)

This paper contains 51 sections, 3 theorems, 57 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Agent Debate for LLM Reasoning
  4. Context Compression for LLMs
  5. Vision-Language Models
  6. Method
  7. Preliminaries and Computational Challenges
  8. Framework Overview
  9. Training Phase
  10. ENCODER PIPELINE
  11. Adapter Network
  12. Training Procedure
  13. Inference Phase
  14. Text-to-Image Rendering
  15. Vision Encoding
  16. ...and 36 more sections

Key Result

Theorem 4.2

Suppose: (1) Compression preserves information with probability $p > 0.5$: $\mathbb{P}(I(\mathcal{C}(H_i); Y_q) \geq I(H_i; Y_q) - \varepsilon) \geq p$; (2) Agents have diverse styles with conditionally independent artifacts $V_i \perp V_j \mid q$; (3) Compression reduces artifacts: $I(\mathcal{C}(H while artifacts vanish: $I(f(\mathcal{C}(\mathcal{H})); V) = O(\gamma K)$. Consequently, $D(\mathca

Figures (3)

  • Figure 1: Visual compression addresses the token inefficiency of multi-agent debate. We compare three paradigms across 5 debate rounds using Qwen2-VL on GSM8K with 3 agents. (a) Token consumption: Text-based MAD accumulates 59.2K tokens by Round 5 due to repeatedly storing debate history, while our visual compression reduces this by 92.2% to only 4.5K tokens. Text with summarization achieves 69.6% reduction to 18.0K tokens. (b) Reasoning accuracy: Despite lower computational cost, visual compression achieves the highest accuracy of 81.3%, outperforming text with summarization and text-based MAD. Note: At Round 5, text-based MAD (the gray curve) exceeds the context window limit. Therefore, the accuracy is measured with truncated debate history.
  • Figure 2: Visual compression framework for multi-agent debate. Top: Text-based debate accumulates token count across rounds. Bottom: Our approach converts debate history into visual representations. We first train a lightweight projector to adapt SAM-CLIP features to the vision embedding space of target MLLMs. During inference, debate history text is encoded as images via SAM, embedded through CLIP, and projected into small amount of vision tokens, achieving token reduction while preserving debate context for subsequent rounds.
  • Figure 3: Scaling analysis of DebateOCR showing it's effectiveness across different agent counts and debate rounds.

Theorems & Definitions (9)

  • Definition 4.1: Information Bottleneck
  • Theorem 4.2
  • proof : Proof sketch
  • Corollary 4.3: Sample Complexity
  • Definition 1.1: Information Bottleneck
  • Definition 1.2: Distance to Bottleneck
  • Theorem 1.5
  • proof
  • proof