Learning Language-Driven Sequence-Level Modal-Invariant Representations for Video-Based Visible-Infrared Person Re-Identification

TL;DR

This work tackles cross-modal video person re-identification (VVI-ReID) by learning sequence-level modal-invariant representations across visible and infrared modalities. It introduces LSMRL, a CLIP-guided framework with three modules—STFL for efficient spatial-temporal modeling, SD for semantic diffusion of shared text prompts, and CMI for bidirectional cross-modal interaction—along with modality-level losses to reinforce alignment. The approach demonstrates strong performance on large-scale datasets, with extensive ablations and visualizations confirming the contributions of each component and the overall efficiency. The study highlights the potential of vision-language priors for robust cross-modal video retrieval and suggests directions for finer-grained alignment and representation decoupling in future work.

Abstract

The core of video-based visible-infrared person re-identification (VVI-ReID) lies in learning sequence-level modal-invariant representations across different modalities. Recent research tends to use modality-shared language prompts generated by CLIP to guide the learning of modal-invariant representations. Despite achieving optimal performance, such methods still face limitations in efficient spatial-temporal modeling, sufficient cross-modal interaction, and explicit modality-level loss guidance. To address these issues, we propose the language-driven sequence-level modal-invariant representation learning (LSMRL) method, which includes spatial-temporal feature learning (STFL) module, semantic diffusion (SD) module and cross-modal interaction (CMI) module. To enable parameter- and computation-efficient spatial-temporal modeling, the STFL module is built upon CLIP with minimal modifications. To achieve sufficient cross-modal interaction and enhance the learning of modal-invariant features, the SD module is proposed to diffuse modality-shared language prompts into visible and infrared features to establish preliminary modal consistency. The CMI module is further developed to leverage bidirectional cross-modal self-attention to eliminate residual modality gaps and refine modal-invariant representations. To explicitly enhance the learning of modal-invariant representations, two modality-level losses are introduced to improve the features' discriminative ability and their generalization to unseen categories. Extensive experiments on large-scale VVI-ReID datasets demonstrate the superiority of LSMRL over AOTA methods.

Figures (8)

• Figure 1: The motivation of this paper: (a) Existing method introduces additional computational overhead when modeling spatial-temporal features, and it struggles to learn robust modal-invariant representations due to the lack of cross-modal interaction and modality-level loss guidance. (b) Our method designs a low-cost STFL module to efficiently model spatial-temporal information, constructs SD and CMI modules to achieve sufficient cross-modal interaction, and utilizes MD and MSEL losses to explicitly enhance cross-modal consistency.
• Figure 2: The architecture of the LSMRL method, which consists of the STFL, SD, and CMI modules. The STFL module is first utilized to efficiently model spatial-temporal information of pedestrian sequences with limited extra computational overhead. Then the SD module is applied to diffuse modality-shared text semantics into RGB and IR features, laying the foundation for modal-invariant feature learning. The CMI module is finally used to enable bidirectional cross-modal feature interaction, further eliminating the modality gap and refining discriminative sequence-level modal-invariant representations.
• Figure 3: Diagram of the STG encoder. In MHSA, the $h$ heads are split into two groups: $k$ temporal heads and $h-k$ spatial heads. Here, $H^{Tem}$ denotes temporal head, $H^{Spa}$ denotes spatial head.
• Figure 4: Diagram of the TPS encoder. Tokens from neighboring frames are shifted along the temporal dimension, and spatial-temporal modeling is achieved through the spatial-only Transformer layer, with the temporal length being manually controllable.
• Figure 5: The distribution of distances and features of the baseline and our approach on the test set of BUPTCampus. (a-b) The distributions of the intra-class distances (in red) and inter-class distances (in green). The larger separation between intra-class and inter-class distances demonstrates improved modality alignment and discrimination. (c-d) Visualization of the corresponding feature space by T-SNE. Each unique color in this visualization represents an identity. Visible and infrared modalities are symbolized as solid circles and triangles, respectively.