Sensing and Understanding the World over Air: A Large Multimodal Model for Mobile Networks

TL;DR

The paper investigates Wireless-native multimodal large models (WMLMs) for next-generation wireless networks, arguing that a GPT-style WMLM can leverage wireless signals as a universal anchor modality to fuse sensing and communication data. It introduces a two-stage training paradigm—cross-modal alignment followed by downstream task adaptation—and demonstrates feasibility with a 2.4B-parameter model trained on real ISAC data from DeepSense 6G, achieving strong cross-scenario generalization and multimodal fusion. Key contributions include formalizing WMLM characteristics, detailing fusion and task-representation strategies, and providing empirical evidence that wireless signals can support universal, scalable information processing in ISAC contexts. The work highlights significant potential for WMLMs to transform network intelligence, while outlining data, explainability, and security challenges to be addressed for real-world deployment.

Abstract

Large models (LMs), such as ChatGPT, have made a significant impact across diverse domains and hold great potential to facilitate the evolution of network intelligence. Wireless-native multi-modal large models (WMLMs) can sense and understand the physical world through multi-modal data, serving as a key enabler that integrates communication, sensing, and intelligence, and thus they can boost various smart services to billions of users. However, research on WMLMs remains in its infancy, and the construction of domain-specific multi-modal large models for wireless networks is still underexplored. In this paper, we outlines the key characteristics of WMLMs and summarizes existing methods, on the basis of which a wireless-native multimodal training paradigm is proposed. Specifically, we constructed a GPT-style WMLM model and trained it on a real-world large-scale dataset, leveraging wireless signals as an anchor modality for contrastive learning. Our approach demonstrates outstanding performance compared with existing small-scale models and large multi-modal models, validating the feasibility of using wireless signals as a universal modality and highlighting WMLM's potential to emerge as a new paradigm for future wireless networks.

Figures (4)

• Figure 1: WMLMs for network. WMLMs integrate sensing, communication, and intelligence, providing networks with versatile intelligent information processing capabilities. WMLMs correlate patterns in multi-modal sensing and communication data, thereby empowering various downstream tasks in multiple scenarios.
• Figure 2: Key characteristics of WMLMs in future networking: spatiotemporality, real-time performance, adaptability, and generalization
• Figure 3: Three-stage construction process for WMLMs: a. Data acquisition from wireless communication networks, b. Cloud-based model construction, and c. Model deployment to Edge networks with fine-tuning. Two-stage training paradigm for WMLMs: ①. Cross-modal alignment via contrastive learning and ②. Downstream task adaptation.
• Figure 4: We conducted comparative experiments between our proposed paradigm and baseline models on the multi-modal beam prediction task, using real-world environmental datasets. (a)-(c) present the system environment and parameter configurations, while (d)-(e) demonstrate the experimental results.