Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification

Zhen Sun, Lei Tan, Yunhang Shen, Chengmao Cai, Xing Sun, Pingyang Dai, Liujuan Cao, Rongrong Ji

TL;DR

FlexiReID addresses the need for flexible cross-modal person re-identification across four modalities (Text, Sketch, Infrared, RGB) by introducing Adaptive Expert Activation Mixture of Experts (AEA-MOE) and Cross-Modal Query Fusion (CMQF). The framework supports seven retrieval modes and uses a CLIP-based dual-encoder backbone to fuse heterogeneous modalities, even when some are missing, via Learnable Embedding Features. An expanded CIRS-PEDES dataset enables unified evaluation across all four modalities, and extensive experiments show state-of-the-art or near-state-of-the-art performance across seven retrieval tasks with strong generalization. This work demonstrates a practical, scalable approach to multimodal Re-ID that can adapt to real-world, heterogeneous data scenarios.

Abstract

Multimodal person re-identification (Re-ID) aims to match pedestrian images across different modalities. However, most existing methods focus on limited cross-modal settings and fail to support arbitrary query-retrieval combinations, hindering practical deployment. We propose FlexiReID, a flexible framework that supports seven retrieval modes across four modalities: rgb, infrared, sketches, and text. FlexiReID introduces an adaptive mixture-of-experts (MoE) mechanism to dynamically integrate diverse modality features and a cross-modal query fusion module to enhance multimodal feature extraction. To facilitate comprehensive evaluation, we construct CIRS-PEDES, a unified dataset extending four popular Re-ID datasets to include all four modalities. Extensive experiments demonstrate that FlexiReID achieves state-of-the-art performance and offers strong generalization in complex scenarios.

FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification

TL;DR

FlexiReID addresses the need for flexible cross-modal person re-identification across four modalities (Text, Sketch, Infrared, RGB) by introducing Adaptive Expert Activation Mixture of Experts (AEA-MOE) and Cross-Modal Query Fusion (CMQF). The framework supports seven retrieval modes and uses a CLIP-based dual-encoder backbone to fuse heterogeneous modalities, even when some are missing, via Learnable Embedding Features. An expanded CIRS-PEDES dataset enables unified evaluation across all four modalities, and extensive experiments show state-of-the-art or near-state-of-the-art performance across seven retrieval tasks with strong generalization. This work demonstrates a practical, scalable approach to multimodal Re-ID that can adapt to real-world, heterogeneous data scenarios.

Abstract

Multimodal person re-identification (Re-ID) aims to match pedestrian images across different modalities. However, most existing methods focus on limited cross-modal settings and fail to support arbitrary query-retrieval combinations, hindering practical deployment. We propose FlexiReID, a flexible framework that supports seven retrieval modes across four modalities: rgb, infrared, sketches, and text. FlexiReID introduces an adaptive mixture-of-experts (MoE) mechanism to dynamically integrate diverse modality features and a cross-modal query fusion module to enhance multimodal feature extraction. To facilitate comprehensive evaluation, we construct CIRS-PEDES, a unified dataset extending four popular Re-ID datasets to include all four modalities. Extensive experiments demonstrate that FlexiReID achieves state-of-the-art performance and offers strong generalization in complex scenarios.
Paper Structure (16 sections, 10 equations, 4 figures, 5 tables)

This paper contains 16 sections, 10 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Pre-Training Models
  4. Cross-Modal Person Re-Identification
  5. Mixture-of-Experts
  6. Proposed Method
  7. Framework
  8. Adaptive Expert Allocation Mixture of Experts (AEA-MOE)
  9. Cross-Modal Query Fusion (CMQF)
  10. Optimization and Inference
  11. Experiments
  12. Experimental Setup
  13. Performance Comparison
  14. Ablation Study
  15. Hyper-Parameter Analysis
  16. ...and 1 more sections

Figures (4)

  • Figure 2: The network architecture of the proposed FlexiReID. All visual modalities share a single visual encoder. The incorporation of AEA-MOE facilitates the efficient processing of heterogeneous modality data. CMQF is employed to seamlessly integrate diverse modality features, while learnable embedding features are utilized to compensate for any missing modalities.
  • Figure 3: Traditional Top-K-based routing mechanisms are limited to selecting a fixed number of experts. In contrast, our proposed adaptive routing mechanism introduces a threshold confidence level, allowing the number of selected experts to be dynamically adjusted based on the input data.
  • Figure 4: Our proposed CMQF Module.
  • Figure 5: Ablation Study on the Number of Experts and Threshold Confidence.