Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FSCA-Net: Feature-Separated Cross-Attention Network for Robust Multi-Dataset Training

Yuehai Chen

TL;DR

FSCA-Net tackles cross-domain crowd counting by explicitly disentangling features into domain-invariant and domain-specific subspaces, then enabling adaptive cross-domain interaction through a cross-attention fusion module. Mutual information harmonization—maximizing DI consistency via a lower-bound MI term and minimizing DS redundancy via a CLUB-based upper bound—drives complementary shared-private representations. Empirical results show state-of-the-art cross-dataset generalization and robust in-domain performance across multiple benchmarks, with ablations confirming the necessity of each component. The approach offers practical robustness for real-world crowd analysis, reducing negative transfer without requiring target-domain annotations during training. Overall, FSCA-Net provides a scalable framework for robust, multi-dataset crowd counting with strong generalization to unseen environments.

Abstract

Crowd counting plays a vital role in public safety, traffic regulation, and smart city management. However, despite the impressive progress achieved by CNN- and Transformer-based models, their performance often deteriorates when applied across diverse environments due to severe domain discrepancies. Direct joint training on multiple datasets, which intuitively should enhance generalization, instead results in negative transfer, as shared and domain-specific representations become entangled. To address this challenge, we propose the Feature Separation and Cross-Attention Network FSCA-Net, a unified framework that explicitly disentangles feature representations into domain-invariant and domain-specific components. A novel cross-attention fusion module adaptively models interactions between these components, ensuring effective knowledge transfer while preserving dataset-specific discriminability. Furthermore, a mutual information optimization objective is introduced to maximize consistency among domain-invariant features and minimize redundancy among domain-specific ones, promoting complementary shared-private representations. Extensive experiments on multiple crowd counting benchmarks demonstrate that FSCA-Net effectively mitigates negative transfer and achieves state-of-the-art cross-dataset generalization, providing a robust and scalable solution for real-world crowd analysis.

FSCA-Net: Feature-Separated Cross-Attention Network for Robust Multi-Dataset Training

TL;DR

FSCA-Net tackles cross-domain crowd counting by explicitly disentangling features into domain-invariant and domain-specific subspaces, then enabling adaptive cross-domain interaction through a cross-attention fusion module. Mutual information harmonization—maximizing DI consistency via a lower-bound MI term and minimizing DS redundancy via a CLUB-based upper bound—drives complementary shared-private representations. Empirical results show state-of-the-art cross-dataset generalization and robust in-domain performance across multiple benchmarks, with ablations confirming the necessity of each component. The approach offers practical robustness for real-world crowd analysis, reducing negative transfer without requiring target-domain annotations during training. Overall, FSCA-Net provides a scalable framework for robust, multi-dataset crowd counting with strong generalization to unseen environments.

Abstract

Crowd counting plays a vital role in public safety, traffic regulation, and smart city management. However, despite the impressive progress achieved by CNN- and Transformer-based models, their performance often deteriorates when applied across diverse environments due to severe domain discrepancies. Direct joint training on multiple datasets, which intuitively should enhance generalization, instead results in negative transfer, as shared and domain-specific representations become entangled. To address this challenge, we propose the Feature Separation and Cross-Attention Network FSCA-Net, a unified framework that explicitly disentangles feature representations into domain-invariant and domain-specific components. A novel cross-attention fusion module adaptively models interactions between these components, ensuring effective knowledge transfer while preserving dataset-specific discriminability. Furthermore, a mutual information optimization objective is introduced to maximize consistency among domain-invariant features and minimize redundancy among domain-specific ones, promoting complementary shared-private representations. Extensive experiments on multiple crowd counting benchmarks demonstrate that FSCA-Net effectively mitigates negative transfer and achieves state-of-the-art cross-dataset generalization, providing a robust and scalable solution for real-world crowd analysis.
Paper Structure (26 sections, 12 equations, 4 figures, 6 tables)

This paper contains 26 sections, 12 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Introduction
  3. Related Work
  4. Crowd Counting
  5. Domain Adaptation
  6. Domain Generalization
  7. Mutual Information in Deep Learning
  8. Proposed Method
  9. Overall Framework
  10. Feature Separation and Fusion
  11. Feature Separation: Decoupling Domain-Invariant and Domain-Specific Features
  12. Mathematical Formulation of Feature Separation
  13. Feature Fusion. Strengthening Multi-Source Feature Correlations via Cross-Attention Mechanism
  14. Mutual Information Minimization and Maximization Losses
  15. Mutual Information Maximization Loss (for $\mathbf{F}_{\text{DI}}$ Optimization)
  16. ...and 11 more sections

Figures (4)

  • Figure 1: t-SNE visualization of features extracted from multiple datasets using a baseline model. The features from different datasets are highly entangled, indicating that domain-invariant crowd representations (e.g., density patterns, head shapes) are mixed with domain-specific artifacts (e.g., background textures, illumination, camera perspectives). This entanglement leads to poor cross-dataset generalization, motivating our proposed feature separation strategy.
  • Figure 2: Overview of FSCA-Net. The backbone extracts base features from multiple datasets, which are decomposed into domain-invariant (DI) and domain-specific (DS) features. A cross-attention mechanism enhances inter-dataset interactions, and DI and DS features are further fused into DS-DI features. All features are refined by the decoder to produce the crowd density map. Training is guided by mutual information optimization and a domain-specific strategy, enabling both cross-domain generalization and intra-domain performance.
  • Figure 3: Visual comparison of crowd counting and density prediction results. (A) Input Images; (B) Ground Truth; (C) BL (M); (D) FSCA-Net+BL.
  • Figure 4: t-SNE visualization of feature distributions: (A) features from a jointly trained original method (unable to separate domain-invariant and domain-specific features); (B) features extracted by the proposed FSCA-Net.