Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WiFo-M$^2$: Plug-and-Play Multi-Modal Sensing via Foundation Model to Empower Wireless Communications

Haotian Zhang, Shijian Gao, Xiang Cheng

TL;DR

WiFo-M2 introduces a foundation-model approach to sensing-aided wireless communications, aiming to extract universal out-of-band channel features from multi-modal data (images and LiDAR) and apply them across diverse transceiver modules. The ContraSoM pre-training strategy aligns sensing representations with CSI using modality-specific augmentations, including diffusion-based LiDAR feature augmentation and temporal feature extrapolation, enabling future CSI-aligned feature generation. Across beam prediction, channel estimation, interpolation, and prediction, WiFo-M2 demonstrates universal gains and strong cross-scenario generalization, while remaining lightweight enough for real-time deployment. This work shifts the design paradigm from task-specific pipelines to a plug-and-play, universal representation that broadens the impact of multi-modal sensing on core wireless tasks and demonstrates practical deployability with modest parameter overhead.

Abstract

The growing adoption of sensor-rich intelligent systems has boosted the use of multi-modal sensing to improve wireless communications. However, traditional methods require extensive manual design of data preprocessing, network architecture, and task-specific fine-tuning, which limits both development scalability and real-world deployment. To address this, we propose WiFo-M$^2$, a foundation model that can be easily plugged into existing deep learning-based transceivers for universal performance gains. To extract generalizable out-of-band (OOB) channel features from multi-modal sensing, we introduce ContraSoM, a contrastive pre-training strategy. Once pre-trained, WiFo-M$^2$ infers future OOB channel features from historical sensor data and strengthens feature robustness via modality-specific data augmentation. Experiments show that WiFo-M$^2$ improves performance across multiple transceiver designs and demonstrates strong generalization to unseen scenarios.

WiFo-M$^2$: Plug-and-Play Multi-Modal Sensing via Foundation Model to Empower Wireless Communications

TL;DR

WiFo-M2 introduces a foundation-model approach to sensing-aided wireless communications, aiming to extract universal out-of-band channel features from multi-modal data (images and LiDAR) and apply them across diverse transceiver modules. The ContraSoM pre-training strategy aligns sensing representations with CSI using modality-specific augmentations, including diffusion-based LiDAR feature augmentation and temporal feature extrapolation, enabling future CSI-aligned feature generation. Across beam prediction, channel estimation, interpolation, and prediction, WiFo-M2 demonstrates universal gains and strong cross-scenario generalization, while remaining lightweight enough for real-time deployment. This work shifts the design paradigm from task-specific pipelines to a plug-and-play, universal representation that broadens the impact of multi-modal sensing on core wireless tasks and demonstrates practical deployability with modest parameter overhead.

Abstract

The growing adoption of sensor-rich intelligent systems has boosted the use of multi-modal sensing to improve wireless communications. However, traditional methods require extensive manual design of data preprocessing, network architecture, and task-specific fine-tuning, which limits both development scalability and real-world deployment. To address this, we propose WiFo-M, a foundation model that can be easily plugged into existing deep learning-based transceivers for universal performance gains. To extract generalizable out-of-band (OOB) channel features from multi-modal sensing, we introduce ContraSoM, a contrastive pre-training strategy. Once pre-trained, WiFo-M infers future OOB channel features from historical sensor data and strengthens feature robustness via modality-specific data augmentation. Experiments show that WiFo-M improves performance across multiple transceiver designs and demonstrates strong generalization to unseen scenarios.
Paper Structure (32 sections, 17 equations, 8 figures, 7 tables)

This paper contains 32 sections, 17 equations, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sensing-Assisted Wireless Communications
  4. Contrastive Learning for Cross-Modal Alignment
  5. System Model, Problem Description, and Challenges
  6. System Model and Problem Description
  7. Challenges
  8. WiFo-M$^2$ Framework: Sensing Processing, Feature Extraction, and Pre-training
  9. Unified Sensing Processing Pipeline of WiFo-M$^2$
  10. Image Processing Pipeline
  11. LiDAR Processing Pipeline
  12. Network Architecture and Working Flow of WiFo-M$^2$
  13. CSI Feature Extraction
  14. Image and LiDAR Feature Extraction
  15. Pre-training Strategy for WiFo-M$^2$: ContraSoM
  16. ...and 17 more sections

Figures (8)

  • Figure 1: Comparison between conventional sensing-optimized communications paradigm and the proposed WiFo-M$^2$ paradigm.
  • Figure 2: Overall framework of WiFo-M$^2$, illustrating the two‑stage pipeline: (left) pre-training using ContraSoM strategy, and (right) inference stage for optimizing transceiver design.
  • Figure 3: Network architecture of WiFo-M$^2$-Img/LiDAR.
  • Figure 4: Temporal feature extrapolation in WiFo-M$^2$-Img.
  • Figure 5: Overview of multi-modal sensory data from all datasets.
  • ...and 3 more figures