Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLIP-based Synergistic Knowledge Transfer for Text-based Person Retrieval

Yating Liu, Yaowei Li, Zimo Liu, Wenming Yang, Yaowei Wang, Qingmin Liao

TL;DR

The paper tackles cross-modal gaps in Text-based Person Retrieval (TPR) under limited data by introducing CSKT, a CLIP-based framework that performs parameter-efficient knowledge transfer through Bidirectional Prompts Transferring (BPT) and Dual Adapters Transferring (DAT). By freezing the CLIP backbone and fine-tuning only about 12M parameters (roughly 7.4% of the full model), CSKT leverages CLIP's pre-trained cross-modal knowledge to achieve strong performance across three datasets. Empirical results show competitive or state-of-the-art performance on CUHK-PEDES, ICFG-PEDES, and RTSPReid with improved efficiency and generalization. This PETL-based approach demonstrates that targeted prompt and adapter mechanisms can enable robust, resource-efficient cross-modal alignment for TPR.

Abstract

Text-based Person Retrieval (TPR) aims to retrieve the target person images given a textual query. The primary challenge lies in bridging the substantial gap between vision and language modalities, especially when dealing with limited large-scale datasets. In this paper, we introduce a CLIP-based Synergistic Knowledge Transfer (CSKT) approach for TPR. Specifically, to explore the CLIP's knowledge on input side, we first propose a Bidirectional Prompts Transferring (BPT) module constructed by text-to-image and image-to-text bidirectional prompts and coupling projections. Secondly, Dual Adapters Transferring (DAT) is designed to transfer knowledge on output side of Multi-Head Attention (MHA) in vision and language. This synergistic two-way collaborative mechanism promotes the early-stage feature fusion and efficiently exploits the existing knowledge of CLIP. CSKT outperforms the state-of-the-art approaches across three benchmark datasets when the training parameters merely account for 7.4% of the entire model, demonstrating its remarkable efficiency, effectiveness and generalization.

CLIP-based Synergistic Knowledge Transfer for Text-based Person Retrieval

TL;DR

The paper tackles cross-modal gaps in Text-based Person Retrieval (TPR) under limited data by introducing CSKT, a CLIP-based framework that performs parameter-efficient knowledge transfer through Bidirectional Prompts Transferring (BPT) and Dual Adapters Transferring (DAT). By freezing the CLIP backbone and fine-tuning only about 12M parameters (roughly 7.4% of the full model), CSKT leverages CLIP's pre-trained cross-modal knowledge to achieve strong performance across three datasets. Empirical results show competitive or state-of-the-art performance on CUHK-PEDES, ICFG-PEDES, and RTSPReid with improved efficiency and generalization. This PETL-based approach demonstrates that targeted prompt and adapter mechanisms can enable robust, resource-efficient cross-modal alignment for TPR.

Abstract

Text-based Person Retrieval (TPR) aims to retrieve the target person images given a textual query. The primary challenge lies in bridging the substantial gap between vision and language modalities, especially when dealing with limited large-scale datasets. In this paper, we introduce a CLIP-based Synergistic Knowledge Transfer (CSKT) approach for TPR. Specifically, to explore the CLIP's knowledge on input side, we first propose a Bidirectional Prompts Transferring (BPT) module constructed by text-to-image and image-to-text bidirectional prompts and coupling projections. Secondly, Dual Adapters Transferring (DAT) is designed to transfer knowledge on output side of Multi-Head Attention (MHA) in vision and language. This synergistic two-way collaborative mechanism promotes the early-stage feature fusion and efficiently exploits the existing knowledge of CLIP. CSKT outperforms the state-of-the-art approaches across three benchmark datasets when the training parameters merely account for 7.4% of the entire model, demonstrating its remarkable efficiency, effectiveness and generalization.
Paper Structure (12 sections, 4 equations, 2 figures, 5 tables)

This paper contains 12 sections, 4 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. METHOD
  3. Framework
  4. Bidirectional Prompts Transferring
  5. Dual Adapters Transferring
  6. Train and Inference
  7. EXPERIMENTS
  8. Datasets and Implementation Details
  9. Comparisons with State-of-the-art Methods
  10. Ablation Studies
  11. CONCLUSION
  12. ACKNOWLEDGMENT

Figures (2)

  • Figure 1: Overview of the proposed CSKT framework. The main backbone of CSKT is depicted in Fig (a), which consists of dual encoders (image and text), and two transfer learning modules (purple components), i.e., BPT and DAT. The specific details of DAT are shown in Fig (b), which includes the adapted MLP of encoder layer in two branches. The two dashed lines represent the inside of the two branches. Only the parameters corresponding to the purple components are trainable, while others are forzen. The SDM depicted in Section 2.4 is utilized as a unique training loss.
  • Figure 2: Ablation on prompt length and prompt depth.