Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Cross-Modal to Mixed-Modal Visible-Infrared Re-Identification

Mahdi Alehdaghi, Rajarshi Bhattacharya, Pourya Shamsolmoali, Rafael M. O. Cruz, Eric Granger

TL;DR

This work addresses visible-infrared person re-identification in realistic mixed-modal galleries by proposing MixER, a disentangled feature learning framework. MixER learns two ID-discriminative representations, modality-erased ($Z^e_m$) for cross-modal matching and modality-related ($Z^r_m$) to enhance intra-modal discrimination, using orthogonal decomposition and mutual-information–based objectives with losses including $ ext{L}_{ ext{m}}$ (modality confusion) and $ ext{L}_{ ext{ymr}}$ (modality-aware). The approach uses a single backbone with three lightweight heads and a fusion mechanism for mixed-modal matching, optimized via a Lagrangian that combines $MI$ terms and an orthogonality constraint $ ext{MI}(Z^e_m; Z^r_m)$, controlled by hyperparameters $oldsymbol{ abla}$; during inference, cross-modal matching relies on $Z^e_m$ while mixed-modal matching uses the fused representation $Z^f_m$. Experiments on SYSU-MM01, RegDB, and LLCM show that MixER achieves state-of-the-art performance across mixed-modal and cross-modal settings with modest computational overhead, demonstrating strong practical potential for real-world surveillance with mixed galleries.

Abstract

Visible-infrared person re-identification (VI-ReID) aims to match individuals across different camera modalities, a critical task in modern surveillance systems. While current VI-ReID methods focus on cross-modality matching, real-world applications often involve mixed galleries containing both V and I images, where state-of-the-art methods show significant performance limitations due to large domain shifts and low discrimination across mixed modalities. This is because gallery images from the same modality may have lower domain gaps but correspond to different identities. This paper introduces a novel mixed-modal ReID setting, where galleries contain data from both modalities. To address the domain shift among inter-modal and low discrimination capacity in intra-modal matching, we propose the Mixed Modality-Erased and -Related (MixER) method. The MixER learning approach disentangles modality-specific and modality-shared identity information through orthogonal decomposition, modality-confusion, and ID-modality-related objectives. MixER enhances feature robustness across modalities, improving cross-modal and mixed-modal settings performance. Our extensive experiments on the SYSU-MM01, RegDB and LLMC datasets indicate that our approach can provide state-of-the-art results using a single backbone, and showcase the flexibility of our approach in mixed gallery applications.

From Cross-Modal to Mixed-Modal Visible-Infrared Re-Identification

TL;DR

This work addresses visible-infrared person re-identification in realistic mixed-modal galleries by proposing MixER, a disentangled feature learning framework. MixER learns two ID-discriminative representations, modality-erased () for cross-modal matching and modality-related () to enhance intra-modal discrimination, using orthogonal decomposition and mutual-information–based objectives with losses including (modality confusion) and (modality-aware). The approach uses a single backbone with three lightweight heads and a fusion mechanism for mixed-modal matching, optimized via a Lagrangian that combines terms and an orthogonality constraint , controlled by hyperparameters ; during inference, cross-modal matching relies on while mixed-modal matching uses the fused representation . Experiments on SYSU-MM01, RegDB, and LLCM show that MixER achieves state-of-the-art performance across mixed-modal and cross-modal settings with modest computational overhead, demonstrating strong practical potential for real-world surveillance with mixed galleries.

Abstract

Visible-infrared person re-identification (VI-ReID) aims to match individuals across different camera modalities, a critical task in modern surveillance systems. While current VI-ReID methods focus on cross-modality matching, real-world applications often involve mixed galleries containing both V and I images, where state-of-the-art methods show significant performance limitations due to large domain shifts and low discrimination across mixed modalities. This is because gallery images from the same modality may have lower domain gaps but correspond to different identities. This paper introduces a novel mixed-modal ReID setting, where galleries contain data from both modalities. To address the domain shift among inter-modal and low discrimination capacity in intra-modal matching, we propose the Mixed Modality-Erased and -Related (MixER) method. The MixER learning approach disentangles modality-specific and modality-shared identity information through orthogonal decomposition, modality-confusion, and ID-modality-related objectives. MixER enhances feature robustness across modalities, improving cross-modal and mixed-modal settings performance. Our extensive experiments on the SYSU-MM01, RegDB and LLMC datasets indicate that our approach can provide state-of-the-art results using a single backbone, and showcase the flexibility of our approach in mixed gallery applications.
Paper Structure (24 sections, 4 theorems, 45 equations, 8 figures, 9 tables)

This paper contains 24 sections, 4 theorems, 45 equations, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed MixER Method
  4. Mutual Information Analysis
  5. Modality Erased and Related Learning
  6. Inference
  7. Experimental Methodology and Results
  8. Comparison with State-of-the-Art Methods
  9. Ablation Studies
  10. Qualitative Analysis
  11. Conclusion
  12. Proofs:
  13. ‌Backgrounds
  14. Minimizing $\text{MI\xspace}(Z^e_{m};M)$
  15. Maximizing $\text{MI\xspace}(Z^e_{m};Y|M)$
  16. ...and 9 more sections

Key Result

Theorem 1

If $Z^e_{m}$ and $Z^r_{m}$ are independent, then $\text{MI\xspace}(Z^e_{m},Z^r_{m};Y)=\text{MI\xspace}(Z^e_{m};Y) + \text{MI\xspace}(Z^r_{m};Y)$.

Figures (8)

  • Figure 1: (a) VI-ReID of a query image matched against a cross-modal and mixed-modal gallery. (b) With VI-ReID methods, V, I, and ID information reveal modality-specific and modality-shared ID features. (c) Mixed-modal approaches leverage these features for matching, while uni-modal methods are limited to intra-modality matching due to a large modality gap, and cross-modal methods enable inter-modal matching by extracting shared features. Our MixER method disentangles these features to reduce the gap in shared features and enhance the discrimination in modality-specific features.
  • Figure 2: The overall architecture of our proposed MixER method. It extracts two independent ID-discriminative feature vectors by orthogonal decomposition, modality-confusion, and modality-aware losses for learning modality-erased and modality-related feature representation.
  • Figure 3: Different settings for forming a gallery based on modality, camera, and query identity. The gallery set images are (a) all images from another modality, (b) all images from both modalities, and all images except ones with the (c) same camera, (d) same camera and same identity, and (e) same identity in the same modality as the query.
  • Figure 4: The intra-class (green) and inter-class (pink) distances distribution of features in different gallery settings. Here SAAI method SAAI has been considered as the Baseline.
  • Figure 5: Rank-6 retrieval results obtained by the baseline (top) and the proposed MixER (bottom) on SYSU dataset under Mix-Cam-ID (left) and Mix-ID (right) settings.
  • ...and 3 more figures

Theorems & Definitions (10)

  • Theorem 1
  • proof
  • Definition 1: Sufficiency
  • Definition 2: Data Processing Inequality
  • Theorem 1
  • proof
  • Proposition 1
  • proof
  • Theorem 2
  • proof