Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WiFi2Cap: Semantic Action Captioning from Wi-Fi CSI via Limb-Level Semantic Alignment

Tzu-Ti Wei, Chu-Yu Huang, Yu-Chee Tseng, Jen-Jee Chen

Abstract

Privacy-preserving semantic understanding of human activities is important for indoor sensing, yet existing Wi-Fi CSI-based systems mainly focus on pose estimation or predefined action classification rather than fine-grained language generation. Mapping CSI to natural-language descriptions remains challenging because of the semantic gap between wireless signals and language and direction-sensitive ambiguities such as left/right limb confusion. We propose WiFi2Cap, a three-stage framework for generating action captions directly from Wi-Fi CSI. A vision-language teacher learns transferable supervision from synchronized video-text pairs, and a CSI student is aligned to the teacher's visual space and text embeddings. To improve direction-sensitive captioning, we introduce a Mirror-Consistency Loss that reduces mirrored-action and left-right ambiguities during cross-modal alignment. A prefix-tuned language model then generates action descriptions from CSI embeddings. We also introduce the WiFi2Cap Dataset, a synchronized CSI-RGB-sentence benchmark for semantic captioning from Wi-Fi signals. Experimental results show that WiFi2Cap consistently outperforms baseline methods on BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE, demonstrating effective privacy-friendly semantic sensing.

WiFi2Cap: Semantic Action Captioning from Wi-Fi CSI via Limb-Level Semantic Alignment

Abstract

Privacy-preserving semantic understanding of human activities is important for indoor sensing, yet existing Wi-Fi CSI-based systems mainly focus on pose estimation or predefined action classification rather than fine-grained language generation. Mapping CSI to natural-language descriptions remains challenging because of the semantic gap between wireless signals and language and direction-sensitive ambiguities such as left/right limb confusion. We propose WiFi2Cap, a three-stage framework for generating action captions directly from Wi-Fi CSI. A vision-language teacher learns transferable supervision from synchronized video-text pairs, and a CSI student is aligned to the teacher's visual space and text embeddings. To improve direction-sensitive captioning, we introduce a Mirror-Consistency Loss that reduces mirrored-action and left-right ambiguities during cross-modal alignment. A prefix-tuned language model then generates action descriptions from CSI embeddings. We also introduce the WiFi2Cap Dataset, a synchronized CSI-RGB-sentence benchmark for semantic captioning from Wi-Fi signals. Experimental results show that WiFi2Cap consistently outperforms baseline methods on BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE, demonstrating effective privacy-friendly semantic sensing.
Paper Structure (33 sections, 8 equations, 4 figures, 6 tables)

This paper contains 33 sections, 8 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RF-Based Human Sensing
  4. Cross-Modal Knowledge Transfer
  5. Contrastive Learning
  6. Prefix/Prompt Tuning for Language Conditioning
  7. WiFi2Cap Dataset
  8. Data Collection
  9. Data Annotation
  10. CSI Preprocessing
  11. WiFi2Cap Framework
  12. Vision--Language Teacher
  13. Vision Branch
  14. Text Branch
  15. Contrastive Objective
  16. ...and 18 more sections

Figures (4)

  • Figure 1: WiFi2Cap framework. (a) Stage 1: Vision--language teacher trained with contrastive learning and Mirror-Consistency. (b) Stage 2: CSI--text student trained via teacher-guided visual alignment and CSI--text contrastive learning with Mirror-Consistency. (c) Stage 3: Prefix-guided language generation.
  • Figure 2: Data acquisition setup and multimodal samples. Left: the physical collection setup, including one transmitter (Tx), three receivers (Rx1--Rx3), the RGB camera, and the $4\times6$ grid of participant positions. Right: synchronized CSI heatmaps, RGB frames, and sentence-level action descriptions.
  • Figure 3: CSI encoder. Dual ResNet-18 backbones encode amplitude and phase inputs, followed by gated fusion and projection to a CSI embedding.
  • Figure 4: Visualization of modality alignment between CSI and text embeddings. Each row denotes a CSI embedding and each column denotes a text embedding; brighter values indicate higher cosine similarity. A clearer diagonal pattern after training indicates stronger CSI--text correspondence.