HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning

TL;DR

HoloLLM tackles robust language-grounded sensing in smart homes by integrating rare sensing modalities with text via a novel Universal Modality-Injection Projector (UMIP) and modality-specific encoders. The approach uses CLIP-based coarse embeddings combined with tailored fine-grained features through iterative coarse-to-fine cross-attention, producing multimodal representations that align with text for use in an LLM. A two-stage training regime (tailored encoder pre-training followed by fine-tuning of UMIP and multimodal tokens) plus a human–VLM data-curation pipeline establishes the first multisensory benchmark for human sensing, with substantial gains over existing MLLMs on action recognition, QA, and captioning across MM-Fi and XRF55. The results indicate that UMIP and tailored encoders are key to discriminability and alignment, enabling real-world, language-informed multisensory embodied intelligence while outlining important avenues for future work in planning and efficiency.

Abstract

Embodied agents operating in smart homes must understand human behavior through diverse sensory inputs and communicate via natural language. While Vision-Language Models (VLMs) have enabled impressive language-grounded perception, their reliance on visual data limits robustness in real-world scenarios with occlusions, poor lighting, or privacy constraints. In this paper, we introduce HoloLLM, a Multimodal Large Language Model (MLLM) that integrates uncommon but powerful sensing modalities, such as LiDAR, infrared, mmWave radar, and WiFi, to enable seamless human perception and reasoning across heterogeneous environments. We address two key challenges: (1) the scarcity of aligned modality-text data for rare sensors, and (2) the heterogeneity of their physical signal representations. To overcome these, we design a Universal Modality-Injection Projector (UMIP) that enhances pre-aligned modality embeddings with fine-grained, text-aligned features from tailored encoders via coarse-to-fine cross-attention without introducing significant alignment overhead. We further introduce a human-VLM collaborative data curation pipeline to generate paired textual annotations for sensing datasets. Extensive experiments on two newly constructed benchmarks show that HoloLLM significantly outperforms existing MLLMs, improving language-grounded human sensing accuracy by up to 30%. This work establishes a new foundation for real-world, language-informed multisensory embodied intelligence.

Figures (13)

• Figure 1: HoloLLM achieves seamless and language-grounded human perception and reasoning with complementary sensing modalities. It overcomes real-world challenges, e.g., illumination and privacy, with superior performance on human action recognition, question answering (QA), and captioning tasks, which enables embodied agents to work intelligently across diverse scenarios.
• Figure 2: Architecture of HoloLLM. Given multimodal inputs $\mathbf{X}^m$, HoloLLM utilizes modality-specific tokenizers and a universal encoder to extract pre-aligned initial embeddings $\mathbf{Y}_{CLIP}^m$. Meanwhile, pre-trained tailored encoders are applied to explore modality features $\mathbf{Y}_{T}^m$. The UMIP then transforms $\mathbf{Y}_{CLIP}^m$ and $\mathbf{Y}_{T}^m$ into coarse queries $\mathbf{Q}^m$ and fine-grained keys and values $\mathbf{K}^m / \mathbf{V}^m$. By iteratively enhancing the queries via coarse-to-fine cross-attention and projecting them to the LLM text space, the aligned multimodal tokens $\mathbf{Z}^m$ fully enriched by modality features can be achieved.
• Figure 3: Comparison between UMIP and other projectors: (a) Modality-Specific Projector xu2024pointllmzhao2023chatbridgegirdhar2023imagebind, (b) Universal Projector han2024onellm, and (c) Universal Modality-Injection Projector (Ours).
• Figure 4: Data curation pipeline for (a) Action question answering (QA) and (b) Action Caption.
• Figure 5: Evaluation of Human Action Recognition on MM-Fi yang2023sensefi and XRF55 wang2024xrf55 across three benchmarks in terms of Accuracy (Better to zoom in).