Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PFM-VEPAR: Prompting Foundation Models for RGB-Event Camera based Pedestrian Attribute Recognition

Minghe Xu, Rouying Wu, ChiaWei Chu, Xiao Wang, Yu Li

Abstract

Event-based pedestrian attribute recognition (PAR) leverages motion cues to enhance RGB cameras in low-light and motion-blur scenarios, enabling more accurate inference of attributes like age and emotion. However, existing two-stream multimodal fusion methods introduce significant computational overhead and neglect the valuable guidance from contextual samples. To address these limitations, this paper proposes an Event Prompter. Discarding the computationally expensive auxiliary backbone, this module directly applies extremely lightweight and efficient Discrete Cosine Transform (DCT) and Inverse DCT (IDCT) operations to the event data. This design extracts frequency-domain event features at a minimal computational cost, thereby effectively augmenting the RGB branch. Furthermore, an external memory bank designed to provide rich prior knowledge, combined with modern Hopfield networks, enables associative memory-augmented representation learning. This mechanism effectively mines and leverages global relational knowledge across different samples. Finally, a cross-attention mechanism fuses the RGB and event modalities, followed by feed-forward networks for attribute prediction. Extensive experiments on multiple benchmark datasets fully validate the effectiveness of the proposed RGB-Event PAR framework. The source code of this paper will be released on https://github.com/Event-AHU/OpenPAR

PFM-VEPAR: Prompting Foundation Models for RGB-Event Camera based Pedestrian Attribute Recognition

Abstract

Event-based pedestrian attribute recognition (PAR) leverages motion cues to enhance RGB cameras in low-light and motion-blur scenarios, enabling more accurate inference of attributes like age and emotion. However, existing two-stream multimodal fusion methods introduce significant computational overhead and neglect the valuable guidance from contextual samples. To address these limitations, this paper proposes an Event Prompter. Discarding the computationally expensive auxiliary backbone, this module directly applies extremely lightweight and efficient Discrete Cosine Transform (DCT) and Inverse DCT (IDCT) operations to the event data. This design extracts frequency-domain event features at a minimal computational cost, thereby effectively augmenting the RGB branch. Furthermore, an external memory bank designed to provide rich prior knowledge, combined with modern Hopfield networks, enables associative memory-augmented representation learning. This mechanism effectively mines and leverages global relational knowledge across different samples. Finally, a cross-attention mechanism fuses the RGB and event modalities, followed by feed-forward networks for attribute prediction. Extensive experiments on multiple benchmark datasets fully validate the effectiveness of the proposed RGB-Event PAR framework. The source code of this paper will be released on https://github.com/Event-AHU/OpenPAR
Paper Structure (18 sections, 12 equations, 5 figures, 5 tables)

This paper contains 18 sections, 12 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Pedestrian Attribute Recognition
  4. Pre-trained Foundation Model
  5. Modern Hopfield Network
  6. Methodology
  7. Overview
  8. PFM-VEPAR Framework
  9. Loss Function
  10. Experiments
  11. Datasets and Evaluation Metric
  12. Implementation Details
  13. Comparison on Public Benchmarks
  14. Component Analysis
  15. Ablation Study
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Comparison between existing RGB-Event PAR models and the newly proposed one.
  • Figure 2: An overview of our proposed RGB-Event pedestrian attribute recognition framework, i.e., PFM-VEPAR. In this paper, we propose a lightweight, frequency-aware Event Prompter that enhances RGB feature learning with minimal computational cost. Given an RGB-Event pair, non-overlapping patches are fed into a ViT backbone and the Event Prompter. An associative memory–enhanced context mining strategy strengthens multi-modal representation, followed by cross-attention fusion and a feed-forward network for attribute prediction.
  • Figure 3: An illustration of the (left) Modern Hopfield Network and (right) Hopfield layers.
  • Figure 4: Component Analysis of PFM-VEPAR.
  • Figure 5: Visualization of some predicted pedestrian attributes using our proposed Model PFM-VEPAR.