Why Keep Your Doubts to Yourself? Trading Visual Uncertainties in Multi-Agent Bandit Systems

TL;DR

This paper addresses the economic viability of scaling Vision-Language Model–based multi-agent systems by reframing coordination as a decentralized market for uncertainty. It introduces Agora, which minted cognitive uncertainty into tradable assets across perceptual, semantic, and inferential dimensions, and uses a profitability-driven trading protocol guided by a market-aware Thompson Sampling broker to reach cost-efficient equilibria. Empirical results across five multimodal benchmarks show Agora achieves up to +8.5% accuracy over baselines on MMMU and attains over 3x cost reduction, with state-of-the-art performance on MMBench, InfoVQA, and CC-OCR, validating the approach's scalability and practicality. The work contributes a formal economic framework for uncertainty management in MAS, demonstrates the superiority of market-based coordination over heuristic proxies, and provides detailed analyses of component ablations, pool configurations, and runtime considerations, underscoring its potential to enable economically viable large-scale visual intelligence systems.

Abstract

Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination. We introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3x. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems.

Figures (7)

• Figure 1: Comparison of heuristic coordination and Agora. Unlike heuristics that rely on flawed proxies, Agora forms a dynamic market for uncertainty, where emergent prices enable coordination.
• Figure 2: Final epistemic uncertainty of Agora (blue, 0.16) vs. KABB-VLM (orange, 0.21).
• Figure 3: In Agora, query uncertainty is split into perceptual ($U_{\text{perc}}$), semantic ($U_{\text{sem}}$), and inferential ($U_{\text{inf}}$). A market-aware broker trades these among agents for efficient resolution.
• Figure 4: Comparison with alternative routing and multi-agent strategies on MMBench_V11_Test (N=6). Lower is better for Cost, Time, COI, and $U_{final\_epis}$; higher is better for Accuracy.
• Figure 5: Cost–Performance vs. Accuracy on MMBench_TEST_V11.The curve illustrates Agora's ability to achieve a superior Pareto frontier. As the agent pool grows, the system improves accuracy at a sub-linear cost increase, highlighting the efficiency of its market-aware MAB.