MDReID: Modality-Decoupled Learning for Any-to-Any Multi-Modal Object Re-Identification

TL;DR

MDReID tackles modality-inconsistent ReID by introducing a flexible any-to-any framework that operates across arbitrary query-gallery modality pairs. It couples Modality-Decoupled Learning (MDL), which explicitly splits features into modality-shared and modality-specific components using per-modality tokens in a ViT backbone, with Modality-aware Metric Learning (MML) that enforces orthogonality and complementarity via $L_{ROL}$ and $L_{KDL}$ losses. The approach yields state-of-the-art results on RGBNT201, RGBNT100, and MSVR310, with significant gains in both modality-matched and modality-mismatched settings, and demonstrates robustness to missing modalities while maintaining computational efficiency. This work offers a practical, scalable solution for any-to-any ReID in real-world multispectral systems, enabling reliable re-identification across diverse sensor configurations.

Abstract

Real-world object re-identification (ReID) systems often face modality inconsistencies, where query and gallery images come from different sensors (e.g., RGB, NIR, TIR). However, most existing methods assume modality-matched conditions, which limits their robustness and scalability in practical applications. To address this challenge, we propose MDReID, a flexible any-to-any image-level ReID framework designed to operate under both modality-matched and modality-mismatched scenarios. MDReID builds on the insight that modality information can be decomposed into two components: modality-shared features that are predictable and transferable, and modality-specific features that capture unique, modality-dependent characteristics. To effectively leverage this, MDReID introduces two key components: the Modality Decoupling Learning (MDL) and Modality-aware Metric Learning (MML). Specifically, MDL explicitly decomposes modality features into modality-shared and modality-specific representations, enabling effective retrieval in both modality-aligned and mismatched scenarios. MML, a tailored metric learning strategy, further enforces orthogonality and complementarity between the two components to enhance discriminative power across modalities. Extensive experiments conducted on three challenging multi-modality ReID benchmarks (RGBNT201, RGBNT100, MSVR310) consistently demonstrate the superiority of MDReID. Notably, MDReID achieves significant mAP improvements of 9.8\%, 3.0\%, and 11.5\% in general modality-matched scenarios, and average gains of 3.4\%, 11.8\%, and 10.9\% in modality-mismatched scenarios, respectively. The code is available at: \textcolor{magenta}{https://github.com/stone96123/MDReID}.

Figures (3)

• Figure 1: Illustration of the motivation of MDReID.(a) Though the availability of spectral modalities (e.g., RGB, NIR, TIR) varies across queries and galleries, recent methods only focus on the modality-matched scenarios, which limits their practical applicability. (b) MDReID overcomes the rigidity of modality constraints by disentangling modality-shared and modality-specific features, enabling effective matching between queries and galleries from arbitrary modalities.
• Figure 2: Overall framework of MDReID. MDReID is designed to support retrieval across arbitrary modality combinations. It disentangles features into shared and specific components to boost performance in both matched and mismatched scenarios. Additionally, by leveraging representation orthogonality loss (ROL) and knowledge discrepancy loss (KDL), MDReID refines feature separation and enhances retrieval robustness.
• Figure 3: The visualization of the ablation study. Triangles, circles, and X symbols correspond to the RGB, NIR, and TIR modalities, respectively. Black borders highlight shared features for clarity. Without ROL, distinguishing between shared and modality-specific features is challenging, but its introduction effectively clusters the shared features.