Hierarchical Prompt Learning for Image- and Text-Based Person Re-Identification

TL;DR

This work tackles the challenge of unified person re-identification with both image and text queries, where joint training often suffers from semantic conflicts between instance-level (text-driven) and identity-level (image-driven) cues. It introduces Hierarchical Prompt Learning (HPL), comprising a Task-Routed Transformer (TRT) with dual class tokens, a Hierarchical Prompt Learning module that merges fixed identity prompts with dynamically generated instance prompts via modality-specific inversion networks, and a Cross-Modal Prompt Regularization (CMPR) to align prompts across modalities. The approach uses a two-stage optimization to construct and then leverage these prompts for cross-modal learning, applying losses for intra- and cross-modal alignment and instance-level regularization. Empirical results on six benchmark combinations show state-of-the-art performance on both I2I and T2I ReID tasks, validating the effectiveness of joint modeling and prompt-based supervision for multi-modal person retrieval. The framework demonstrates strong cross-modal coherence, robust identity discrimination, and practical potential for unified multi-modal ReID systems, with clear gains from TRT, HPL, and CMPR components.

Abstract

Person re-identification (ReID) aims to retrieve target pedestrian images given either visual queries (image-to-image, I2I) or textual descriptions (text-to-image, T2I). Although both tasks share a common retrieval objective, they pose distinct challenges: I2I emphasizes discriminative identity learning, while T2I requires accurate cross-modal semantic alignment. Existing methods often treat these tasks separately, which may lead to representation entanglement and suboptimal performance. To address this, we propose a unified framework named Hierarchical Prompt Learning (HPL), which leverages task-aware prompt modeling to jointly optimize both tasks. Specifically, we first introduce a Task-Routed Transformer, which incorporates dual classification tokens into a shared visual encoder to route features for I2I and T2I branches respectively. On top of this, we develop a hierarchical prompt generation scheme that integrates identity-level learnable tokens with instance-level pseudo-text tokens. These pseudo-tokens are derived from image or text features via modality-specific inversion networks, injecting fine-grained, instance-specific semantics into the prompts. Furthermore, we propose a Cross-Modal Prompt Regularization strategy to enforce semantic alignment in the prompt token space, ensuring that pseudo-prompts preserve source-modality characteristics while enhancing cross-modal transferability. Extensive experiments on multiple ReID benchmarks validate the effectiveness of our method, achieving state-of-the-art performance on both I2I and T2I tasks.

Figures (4)

• Figure 1: (a) Performance degradation occurs when jointly training I2I and T2I ReID tasks in a single model, compared to training each task independently. (b) The underlying cause lies in the semantic conflict: T2I emphasizes instance-specific attributes (e.g., "holding a table") highlighted in text but ignored by I2I, despite shared identity-level cues such as clothing and gender.
• Figure 2: Overview of our framework. The framework comprises three core modules: (1) Task-Routed Transformer (TRT), which introduces dual classification tokens into the shared visual encoder to enable task-specific feature learning for I2I and T2I tasks; (2) Hierarchical Prompt Learning (HPL), which constructs and aligns identity-level and instance-level prompts via modality-specific inversion networks; (3) Cross-Modal Prompt Regularization (CMPR), which enforces semantic consistency between visual- and text-derived prompts at the instance level. Arrows indicate the directional flow of information and interactions among modules.
• Figure 3: Effect of $\lambda_1$ and $\lambda_2$ on Rank-1 accuracy on CUHK-PEDES + Market1501.
• Figure 4: Grad-CAM comparison of I2I and T2I tokens, showing stronger focus on view-specific cues in T2I aligned with text descriptions.