Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study of Mamba-based Pedestrian Attribute Recognition

Xiao Wang, Weizhe Kong, Jiandong Jin, Shiao Wang, Ruichong Gao, Qingchuan Ma, Chenglong Li, Jin Tang

TL;DR

This paper empirically evaluates Mamba-based architectures for Pedestrian Attribute Recognition (PAR) as a more efficient alternative to Transformer-heavy models. It designs two PAR frameworks—image-based and image-text fusion—around Mamba backbones (Vim/VMamba) supplemented by a text encoder and a Vision-Semantic Fusion module, and it explores eight hybrid Mamba-Transformer variants. Across PA100K, PETA, RAP, WIDER, MSP60K, and zero-shot splits, Vim-based fusion can boost performance in some settings, while VMamba interactions with text can be inconsistent; several hybrids (notably MaHDFT) achieve competitive results against ViT baselines, with KDTM distillation offering gains in some configurations. The work also analyzes efficiency, memory usage, and ablations to provide guidance for designing efficient Mamba-based PAR and multi-label recognition systems, highlighting both potential and limitations for future improvements.

Abstract

Current strong pedestrian attribute recognition models are developed based on Transformer networks, which are computationally heavy. Recently proposed models with linear complexity (e.g., Mamba) have garnered significant attention and have achieved a good balance between accuracy and computational cost across a variety of visual tasks. Relevant review articles also suggest that while these models can perform well on some pedestrian attribute recognition datasets, they are generally weaker than the corresponding Transformer models. To further tap into the potential of the novel Mamba architecture for PAR tasks, this paper designs and adapts Mamba into two typical PAR frameworks, i.e., the text-image fusion approach and pure vision Mamba multi-label recognition framework. It is found that interacting with attribute tags as additional input does not always lead to an improvement, specifically, Vim can be enhanced, but VMamba cannot. This paper further designs various hybrid Mamba-Transformer variants and conducts thorough experimental validations. These experimental results indicate that simply enhancing Mamba with a Transformer does not always lead to performance improvements but yields better results under certain settings. We hope this empirical study can further inspire research in Mamba for PAR, and even extend into the domain of multi-label recognition, through the design of these network structures and comprehensive experimentation. The source code of this work will be released at \url{https://github.com/Event-AHU/OpenPAR}

An Empirical Study of Mamba-based Pedestrian Attribute Recognition

TL;DR

This paper empirically evaluates Mamba-based architectures for Pedestrian Attribute Recognition (PAR) as a more efficient alternative to Transformer-heavy models. It designs two PAR frameworks—image-based and image-text fusion—around Mamba backbones (Vim/VMamba) supplemented by a text encoder and a Vision-Semantic Fusion module, and it explores eight hybrid Mamba-Transformer variants. Across PA100K, PETA, RAP, WIDER, MSP60K, and zero-shot splits, Vim-based fusion can boost performance in some settings, while VMamba interactions with text can be inconsistent; several hybrids (notably MaHDFT) achieve competitive results against ViT baselines, with KDTM distillation offering gains in some configurations. The work also analyzes efficiency, memory usage, and ablations to provide guidance for designing efficient Mamba-based PAR and multi-label recognition systems, highlighting both potential and limitations for future improvements.

Abstract

Current strong pedestrian attribute recognition models are developed based on Transformer networks, which are computationally heavy. Recently proposed models with linear complexity (e.g., Mamba) have garnered significant attention and have achieved a good balance between accuracy and computational cost across a variety of visual tasks. Relevant review articles also suggest that while these models can perform well on some pedestrian attribute recognition datasets, they are generally weaker than the corresponding Transformer models. To further tap into the potential of the novel Mamba architecture for PAR tasks, this paper designs and adapts Mamba into two typical PAR frameworks, i.e., the text-image fusion approach and pure vision Mamba multi-label recognition framework. It is found that interacting with attribute tags as additional input does not always lead to an improvement, specifically, Vim can be enhanced, but VMamba cannot. This paper further designs various hybrid Mamba-Transformer variants and conducts thorough experimental validations. These experimental results indicate that simply enhancing Mamba with a Transformer does not always lead to performance improvements but yields better results under certain settings. We hope this empirical study can further inspire research in Mamba for PAR, and even extend into the domain of multi-label recognition, through the design of these network structures and comprehensive experimentation. The source code of this work will be released at \url{https://github.com/Event-AHU/OpenPAR}
Paper Structure (18 sections, 10 equations, 5 figures, 7 tables)

This paper contains 18 sections, 10 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Pedestrian Attribute Recognition
  4. State Space Model
  5. Our Proposed Approach
  6. Preliminary: State Space Model
  7. When Mamba Meets PAR
  8. Hybrid Mamba-Transformer for PAR
  9. Experiments
  10. Datasets and Evaluation Metric
  11. Implementation Details
  12. Comparison on Public PAR Benchmarks
  13. Ablation Study
  14. Efficiency Analysis
  15. Comparison of Hybrid Mamba-Transformer
  16. ...and 3 more sections

Figures (5)

  • Figure 1: An illustration of our proposed Mamba-based pedestrian attribute recognition framework.
  • Figure 2: An illustration of our proposed vision-language fusion framework based on Mamba for pedestrian attribute recognition.
  • Figure 3: An illustration of various hybrid Mamba-Transformer frameworks for PAR.
  • Figure 4: Comparison on (a) model parameters, (b) Time cost, and (c) GPU cost.
  • Figure 5: Visualization of the predicted human attributes using our Mamba-based PAR framework.