Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DCG ReID: Disentangling Collaboration and Guidance Fusion Representations for Multi-modal Vehicle Re-Identification

Aihua Zheng, Ya Gao, Shihao Li, Chenglong Li, Jin Tang

TL;DR

DCG-ReID tackles multi-modal vehicle ReID with RGB, NIR, and TIR by addressing modality quality uncertainty that causes fusion conflicts. It introduces Dynamic Confidence-based Disentangling Weighting (DCDW) to compute per-modality confidence and splits fusion into Collaboration Fusion Module (CFM) for balanced data and Guidance Fusion Module (GFM) for unbalanced data, plus an Inter-Mining mechanism for cross-modal interaction. The approach leverages a shared CLIP-based encoder and jointly optimizes identity, triplet, and CLIP-based alignment losses to produce robust fused representations. Experiments on WMVeID863, MSVR310, and RGBNT100 demonstrate state-of-the-art performance and robustness to missing modalities, with code to be released upon acceptance.

Abstract

Multi-modal vehicle Re-Identification (ReID) aims to leverage complementary information from RGB, Near Infrared (NIR), and Thermal Infrared (TIR) modalities to retrieve the same vehicle. The challenges of multi-modal vehicle ReID arise from the uncertainty of modality quality distribution induced by inherent discrepancies across modalities, resulting in distinct conflicting fusion requirements for data with balanced and unbalanced quality distributions. Existing methods handle all multi-modal data within a single fusion model, overlooking the different needs of the two data types and making it difficult to decouple the conflict between intra-class consistency and inter-modal heterogeneity. To this end, we propose Disentangle Collaboration and Guidance Fusion Representations for Multi-modal Vehicle ReID (DCG-ReID). Specifically, to disentangle heterogeneous quality-distributed modal data without mutual interference, we first design the Dynamic Confidence-based Disentangling Weighting (DCDW) mechanism: dynamically reweighting three-modal contributions via interaction-derived modal confidence to build a disentangled fusion framework. Building on DCDW, we develop two scenario-specific fusion strategies: (1) for balanced quality distributions, Collaboration Fusion Module (CFM) mines pairwise consensus features to capture shared discriminative information and boost intra-class consistency; (2) for unbalanced distributions, Guidance Fusion Module (GFM) implements differential amplification of modal discriminative disparities to reinforce dominant modality advantages, guide auxiliary modalities to mine complementary discriminative info, and mitigate inter-modal divergence to boost multi-modal joint decision performance. Extensive experiments on three multi-modal ReID benchmarks (WMVeID863, MSVR310, RGBNT100) validate the effectiveness of our method. Code will be released upon acceptance.

DCG ReID: Disentangling Collaboration and Guidance Fusion Representations for Multi-modal Vehicle Re-Identification

TL;DR

DCG-ReID tackles multi-modal vehicle ReID with RGB, NIR, and TIR by addressing modality quality uncertainty that causes fusion conflicts. It introduces Dynamic Confidence-based Disentangling Weighting (DCDW) to compute per-modality confidence and splits fusion into Collaboration Fusion Module (CFM) for balanced data and Guidance Fusion Module (GFM) for unbalanced data, plus an Inter-Mining mechanism for cross-modal interaction. The approach leverages a shared CLIP-based encoder and jointly optimizes identity, triplet, and CLIP-based alignment losses to produce robust fused representations. Experiments on WMVeID863, MSVR310, and RGBNT100 demonstrate state-of-the-art performance and robustness to missing modalities, with code to be released upon acceptance.

Abstract

Multi-modal vehicle Re-Identification (ReID) aims to leverage complementary information from RGB, Near Infrared (NIR), and Thermal Infrared (TIR) modalities to retrieve the same vehicle. The challenges of multi-modal vehicle ReID arise from the uncertainty of modality quality distribution induced by inherent discrepancies across modalities, resulting in distinct conflicting fusion requirements for data with balanced and unbalanced quality distributions. Existing methods handle all multi-modal data within a single fusion model, overlooking the different needs of the two data types and making it difficult to decouple the conflict between intra-class consistency and inter-modal heterogeneity. To this end, we propose Disentangle Collaboration and Guidance Fusion Representations for Multi-modal Vehicle ReID (DCG-ReID). Specifically, to disentangle heterogeneous quality-distributed modal data without mutual interference, we first design the Dynamic Confidence-based Disentangling Weighting (DCDW) mechanism: dynamically reweighting three-modal contributions via interaction-derived modal confidence to build a disentangled fusion framework. Building on DCDW, we develop two scenario-specific fusion strategies: (1) for balanced quality distributions, Collaboration Fusion Module (CFM) mines pairwise consensus features to capture shared discriminative information and boost intra-class consistency; (2) for unbalanced distributions, Guidance Fusion Module (GFM) implements differential amplification of modal discriminative disparities to reinforce dominant modality advantages, guide auxiliary modalities to mine complementary discriminative info, and mitigate inter-modal divergence to boost multi-modal joint decision performance. Extensive experiments on three multi-modal ReID benchmarks (WMVeID863, MSVR310, RGBNT100) validate the effectiveness of our method. Code will be released upon acceptance.
Paper Structure (18 sections, 24 equations, 7 figures, 5 tables)

This paper contains 18 sections, 24 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. Multi-modal Vehicle ReID
  4. Uncertainty in Deep Learning
  5. METHODOLOGY
  6. Feature Extraction
  7. Dynamic Confidence-based Disentangling Weighting
  8. Collaboration Fusion Module
  9. Guidance Fusion Module
  10. Objective Function
  11. EXPERIMENT
  12. Datasets and Evaluation Protocols
  13. Implementation Details
  14. Comparison with State-of-the-art Methods
  15. Ablation Study
  16. ...and 3 more sections

Figures (7)

  • Figure 1: (a) Inherent key challenges in multi-modal vehicle ReID lead to unbalanced multi-modal quality distribution. (b) The heatmap of AP and Rank-1 gains on samples between our fusion decoupling method and the unified fusion model (i.e., it does not distinguish between the quality distributions of multi-modal samples); The basic approach of our DCG based on fusion decoupling.
  • Figure 2: Our reliable multi-modal dynamic weighting vs. single-modal weighting.
  • Figure 3: The framework of our DCG. Firstly, the three-modal features (RGB, NIR, TIR) are extracted by the shared CLIP visual encoder. They then enter our designed Dynamic Confidence-based Disentangling Weighting (DCDW) to obtain dynamic confidence-based weights. According to the weight distribution, we introduce two fusion strategies, namely the Collaboration Fusion Module (CFM) and the Guidance Fusion Module (GFM), which are based on the reliable weights of the three modalities. Finally, we fuse these features to generate robust multi-modal features
  • Figure 4: Comparison of WMVeID863 performance on samples between our decoupling fusion method and unified fusion method (feeding all samples into two fusion strategies).
  • Figure 5: Feature distributions with t-SNE tsne on WMVeID863. Different colors represent different identities. (a) Baseline, (b) Baseline + CFM, (c) Baseline + GFM, (d) DCG. Better view with colors and zooming in.
  • ...and 2 more figures