Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Transformer-based Multimodal Fusion Model for Efficient Crowd Counting Using Visual and Wireless Signals

Zhe Cui, Yuli Li, Le-Nam Tran

TL;DR

This work addresses crowd counting under challenging conditions by combining visual images with WiFi CSI in a unified Transformer-based fusion framework. TransFusion uses cross-modal attention with a linear-transformer backbone and a multi-scale CNN to capture global context and local details from both modalities, achieving state-of-the-art accuracy with lower computation than standard Transformers. Key contributions include a detailed cross-modal fusion architecture, Hampel-filter-based CSI denoising, a temporal convolution layer, and an ablation study demonstrating substantial performance gains from each component. The approach offers practical value for device-free, privacy-preserving crowd analytics in indoor environments, with strong generalization and efficiency benefits.

Abstract

Current crowd-counting models often rely on single-modal inputs, such as visual images or wireless signal data, which can result in significant information loss and suboptimal recognition performance. To address these shortcomings, we propose TransFusion, a novel multimodal fusion-based crowd-counting model that integrates Channel State Information (CSI) with image data. By leveraging the powerful capabilities of Transformer networks, TransFusion effectively combines these two distinct data modalities, enabling the capture of comprehensive global contextual information that is critical for accurate crowd estimation. However, while transformers are well capable of capturing global features, they potentially fail to identify finer-grained, local details essential for precise crowd counting. To mitigate this, we incorporate Convolutional Neural Networks (CNNs) into the model architecture, enhancing its ability to extract detailed local features that complement the global context provided by the Transformer. Extensive experimental evaluations demonstrate that TransFusion achieves high accuracy with minimal counting errors while maintaining superior efficiency.

A Transformer-based Multimodal Fusion Model for Efficient Crowd Counting Using Visual and Wireless Signals

TL;DR

This work addresses crowd counting under challenging conditions by combining visual images with WiFi CSI in a unified Transformer-based fusion framework. TransFusion uses cross-modal attention with a linear-transformer backbone and a multi-scale CNN to capture global context and local details from both modalities, achieving state-of-the-art accuracy with lower computation than standard Transformers. Key contributions include a detailed cross-modal fusion architecture, Hampel-filter-based CSI denoising, a temporal convolution layer, and an ablation study demonstrating substantial performance gains from each component. The approach offers practical value for device-free, privacy-preserving crowd analytics in indoor environments, with strong generalization and efficiency benefits.

Abstract

Current crowd-counting models often rely on single-modal inputs, such as visual images or wireless signal data, which can result in significant information loss and suboptimal recognition performance. To address these shortcomings, we propose TransFusion, a novel multimodal fusion-based crowd-counting model that integrates Channel State Information (CSI) with image data. By leveraging the powerful capabilities of Transformer networks, TransFusion effectively combines these two distinct data modalities, enabling the capture of comprehensive global contextual information that is critical for accurate crowd estimation. However, while transformers are well capable of capturing global features, they potentially fail to identify finer-grained, local details essential for precise crowd counting. To mitigate this, we incorporate Convolutional Neural Networks (CNNs) into the model architecture, enhancing its ability to extract detailed local features that complement the global context provided by the Transformer. Extensive experimental evaluations demonstrate that TransFusion achieves high accuracy with minimal counting errors while maintaining superior efficiency.
Paper Structure (23 sections, 9 equations, 3 figures, 2 tables)

This paper contains 23 sections, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. System Overview
  4. Data Preprocessing
  5. Preprocessing
  6. Temporal Convolution Layer
  7. Position Encoding
  8. Cross-Modal Feature Fusion with Linear Attention
  9. Multi-Head Linear Attention
  10. Multi-scale CNN
  11. The Feed-forward module
  12. Self-Attention Module
  13. Loss Function
  14. Experiments and evaluations
  15. Experimental settings
  16. ...and 8 more sections

Figures (3)

  • Figure 1: The framework of the proposed TransFusion
  • Figure 2: The layout of experimental environment
  • Figure 3: State of people during experiment