Dual Latent Memory for Visual Multi-agent System

TL;DR

The paper tackles the scaling wall in Visual Multi-Agent Systems caused by information bottlenecks in text-based inter-agent communication. It introduces L$^{2}$-VMAS, which decouples perception and thinking into dual latent memories $M^{P}$ and $M^{T}$, enabling memory synthesis and proactive memory orchestration guided by entropy-based triggers. Across diverse backbones, model sizes, and agent topologies, it achieves stable accuracy gains of $2.7$–$5.4$% while cutting token usage by $21.3$–$44.8$%, demonstrating superior scalability and robustness on both standard and augmented visual benchmarks. The approach offers a model-agnostic framework for scalable VMAS that reduces information loss, improves decision quality, and lowers computational costs in multi-turn visual reasoning tasks.

Abstract

While Visual Multi-Agent Systems (VMAS) promise to enhance comprehensive abilities through inter-agent collaboration, empirical evidence reveals a counter-intuitive "scaling wall": increasing agent turns often degrades performance while exponentially inflating token costs. We attribute this failure to the information bottleneck inherent in text-centric communication, where converting perceptual and thinking trajectories into discrete natural language inevitably induces semantic loss. To this end, we propose L$^{2}$-VMAS, a novel model-agnostic framework that enables inter-agent collaboration with dual latent memories. Furthermore, we decouple the perception and thinking while dynamically synthesizing dual latent memories. Additionally, we introduce an entropy-driven proactive triggering that replaces passive information transmission with efficient, on-demand memory access. Extensive experiments among backbones, sizes, and multi-agent structures demonstrate that our method effectively breaks the "scaling wall" with superb scalability, improving average accuracy by 2.7-5.4% while reducing token usage by 21.3-44.8%. Codes: https://github.com/YU-deep/L2-VMAS.

Figures (10)

• Figure 1: Comparison with existing text-based information transmission, and our proposed dual latent memory tailored for VMAS.
• Figure 2: Accuracy and total token usage among agent turns.
• Figure 3: Comparison of inter-agent transmission content.
• Figure 4: The overview of our proposed L$^{2}$-VMAS.
• Figure 5: Effectiveness Analyses of the dual memory system based on perception, thinking, and mixed tasks.