The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

TL;DR

The Vision Wormhole addresses the token-level bottleneck in heterogeneous MAS by repurposing Vision-Language Model visual encoders as a universal, continuous communication channel. It introduces a Universal Latent Space and hub-and-spoke affine alignment to achieve $O(N)$ scalability, paired with a label-free distillation objective to align the fast visual channel with robust text-based reasoning. The method injects latent content as a visual prompt within the VLM, maintaining bounded bandwidth and avoiding pairwise translators, and demonstrates substantial wall-clock speedups with comparable reasoning fidelity across diverse model families. A weakly supervised codec variant shows the approach is data-efficient, enabling practical deployment with minimal annotation. Overall, Vision Wormhole offers a modular, scalable, and efficient pathway for cross-model collaboration in real-world MAS deployments.

Abstract

Multi-Agent Systems (MAS) powered by Large Language Models have unlocked advanced collaborative reasoning, yet they remain shackled by the inefficiency of discrete text communication, which imposes significant runtime overhead and information quantization loss. While latent state transfer offers a high-bandwidth alternative, existing approaches either assume homogeneous sender-receiver architectures or rely on pair-specific learned translators, limiting scalability and modularity across diverse model families with disjoint manifolds. In this work, we propose the Vision Wormhole, a novel framework that repurposes the visual interface of Vision-Language Models (VLMs) to enable model-agnostic, text-free communication. By introducing a Universal Visual Codec, we map heterogeneous reasoning traces into a shared continuous latent space and inject them directly into the receiver's visual pathway, effectively treating the vision encoder as a universal port for inter-agent telepathy. Our framework adopts a hub-and-spoke topology to reduce pairwise alignment complexity from O(N^2) to O(N) and leverages a label-free, teacher-student distillation objective to align the high-speed visual channel with the robust reasoning patterns of the text pathway. Extensive experiments across heterogeneous model families (e.g., Qwen-VL, Gemma) demonstrate that the Vision Wormhole reduces end-to-end wall-clock time in controlled comparisons while maintaining reasoning fidelity comparable to standard text-based MAS. Code is available at https://github.com/xz-liu/heterogeneous-latent-mas

Figures (10)

• Figure 1: The Vision Wormhole: We repurpose the visual interface of Vision-Language Models (VLMs) to enable text-free communication across heterogeneous Multi-Agent Systems (MAS).
• Figure 2: Vision Wormhole overview. Each agent 1) extracts a bounded latent rollout, encodes it into a fixed-size set of universal tokens, 2) aligns tokens through a shared reference space, and 3) decodes them into an injected perturbation written into the receiver's vision-token span.
• Figure 3: Distribution panels (histograms; log-scaled x-axis) of end-to-end wall-clock time (seconds/query, batch-normalized) for TextMAS vs VW (Blue=TextMAS, Orange=VW).
• Figure 4: Distribution panels (histograms; log-scaled x-axis) of end-to-end wall-clock time (seconds/query, batch-normalized) for TextMAS vs VW (Blue=TextMAS, Orange=VW).
• Figure 5: Pooled boxplot panels of end-to-end wall-clock time after pooled per-cell z-scoring (seconds/query, batch-normalized; Blue=TextMAS, Orange=VW).