ConSensus: Multi-Agent Collaboration for Multimodal Sensing

TL;DR

This work introduces ConSensus, a training-free multi-agent collaboration framework that decomposes multimodal sensing tasks into specialized, modality-aware agents and proposes a hybrid fusion mechanism that balances semantic aggregation, which enables cross-modal reasoning and contextual understanding, with statistical consensus, which provides robustness through agreement across modalities.

Abstract

Large language models (LLMs) are increasingly grounded in sensor data to perceive and reason about human physiology and the physical world. However, accurately interpreting heterogeneous multimodal sensor data remains a fundamental challenge. We show that a single monolithic LLM often fails to reason coherently across modalities, leading to incomplete interpretations and prior-knowledge bias. We introduce ConSensus, a training-free multi-agent collaboration framework that decomposes multimodal sensing tasks into specialized, modality-aware agents. To aggregate agent-level interpretations, we propose a hybrid fusion mechanism that balances semantic aggregation, which enables cross-modal reasoning and contextual understanding, with statistical consensus, which provides robustness through agreement across modalities. While each approach has complementary failure modes, their combination enables reliable inference under sensor noise and missing data. We evaluate ConSensus on five diverse multimodal sensing benchmarks, demonstrating an average accuracy improvement of 7.1% over the single-agent baseline. Furthermore, ConSensus matches or exceeds the performance of iterative multi-agent debate methods while achieving a 12.7 times reduction in average fusion token cost through a single-round hybrid fusion protocol, yielding a robust and efficient solution for real-world multimodal sensing tasks.

Figures (6)

• Figure 1: Illustration of ConSensus. Modality-aware agents provide specialized interpretations aggregated via hybrid fusion for multimodal sensing.
• Figure 2: ConSensus (top-left) achieves higher accuracy (y-axis) at lower cost (x-axis) compared to baselines.
• Figure 3: Examples of LLM-based multimodal sensing on WESAD wesad using gpt-oss-20B.
• Figure 4: Overview of ConSensus: (i) Modality agents generate specialized per-sensor interpretations; (ii) a semantic fusion agent aggregates cross-modal reasoning; (iii) a statistical fusion agent provides an output that anchors the reasoning to the majority; and (iv) a hybrid fusion agent reconciles both outputs to yield the final decision.
• Figure 5: Average input tokens per inference across datasets. Gray bars denote tokens required for initial interpretation, and blue bars denote aggregation or refinement tokens, segmented by rounds.