Babel: A Scalable Pre-trained Model for Multi-Modal Sensing via Expandable Modality Alignment

TL;DR

Babel tackles the data scarcity challenge in multi-modal sensing by proposing expandable modality alignment, converting N-modality alignment into a sequence of binary alignments using pre-trained modality towers, a prototype network, and adaptive growth strategies. The approach enables adding new modalities with minimal retraining, demonstrated by pre-training on six modalities (Wi-Fi, mmWave, IMU, LiDAR, video, depth) and strong zero-shot HAR performance across eight datasets, with notable gains in single- and multi-modal fusion. Key contributions include the pre-trained modality towers, the expandable network architecture with a prototype network, and an adaptive training strategy that dynamically balances modalities during growth. Babel also showcases practical capabilities in cross-modality retrieval and bridging sensing with LLMs, highlighting its potential as a foundation model for scalable, multi-modal sensing in real-world applications.

Abstract

This paper presents Babel, the expandable modality alignment model, specially designed for multi-modal sensing. While there has been considerable work on multi-modality alignment, they all struggle to effectively incorporate multiple sensing modalities due to the data scarcity constraints. How to utilize multi-modal data with partial pairings in sensing remains an unresolved challenge. Babel tackles this challenge by introducing the concept of expandable modality alignment. The key idea involves transforming the N-modality alignment into a series of binary-modality alignments. Novel techniques are also proposed to further mitigate data scarcity issue and balance the contribution of the newly incorporated modality with the previously established modality alignment during the expandable alignment process. We provide the comprehensive implementation. In the pre-training phase, Babel currently aligns 6 sensing modalities, namely Wi-Fi, mmWave, IMU, LiDAR, video, and depth. For the deployment phase, as a foundation model, any single or combination of aligned modalities could be selected from Babel and applied to downstream tasks. Evaluation demonstrates Babel's outstanding performance on eight human activity recognition datasets, compared to a broad range of baselines e.g., the SOTA single-modal sensing networks, multi-modal sensing framework, and multi-modal large language models. Babel not only improves the performance of individual modality sensing (12% averaged accuracy improvement), but also effectively fuses multiple available modalities (up to 22% accuracy increase). Case studies also highlight emerging application scenarios empowered by Babel, including cross-modality retrieval (i.e., sensing imaging), and bridging LLM for sensing comprehension.

Figures (10)

• Figure 1: Five public sensing datasets, XRF55 10.1145/3643543, OPERANet bocus2021operanet, MM-Fi yang2023mmfi, UTD-MHAD chen2015utd and Kinectics-400 kay2017kinetics with binary-paired data cover six modalities.
• Figure 2: Overview of Babel.
• Figure 3: The alignment of two modality towers with pre-trained encoders and concept alignment modules, where $e^m_{\alpha}$ denotes the $m$-th embedding of modality $\alpha$.
• Figure 4: The alignment of multiple modalities with the prototype network in the expandable network architecture.
• Figure 5: t-SNE representations of three modalities obtained by different modality alignment approaches.