Chat-Driven Text Generation and Interaction for Person Retrieval

TL;DR

TBPS relies on textual captions, but manual annotation is costly. This work introduces CTGI, a plug-and-play annotation-free framework with MTG for generating dense pseudo-labels via multi-turn dialogue and MTI for inference-time query refinement using MLLMs, plus a Reconstructor to synthesize captions. It extends CLIP's input capacity with a long-text positional embedding stretching to encode richer descriptions. Experiments on CUHK-PEDES, ICFG-PEDES, and RSTPReid show CTGI achieves competitive or superior performance compared with supervised baselines while eliminating manual captions, enabling scalable TBPS deployment.

Abstract

Text-based person search (TBPS) enables the retrieval of person images from large-scale databases using natural language descriptions, offering critical value in surveillance applications. However, a major challenge lies in the labor-intensive process of obtaining high-quality textual annotations, which limits scalability and practical deployment. To address this, we introduce two complementary modules: Multi-Turn Text Generation (MTG) and Multi-Turn Text Interaction (MTI). MTG generates rich pseudo-labels through simulated dialogues with MLLMs, producing fine-grained and diverse visual descriptions without manual supervision. MTI refines user queries at inference time through dynamic, dialogue-based reasoning, enabling the system to interpret and resolve vague, incomplete, or ambiguous descriptions - characteristics often seen in real-world search scenarios. Together, MTG and MTI form a unified and annotation-free framework that significantly improves retrieval accuracy, robustness, and usability. Extensive evaluations demonstrate that our method achieves competitive or superior results while eliminating the need for manual captions, paving the way for scalable and practical deployment of TBPS systems.

Figures (3)

• Figure 1: Comparison of person description strategies. (a) Human-written captions are concise but often lack compositional depth and attribute coverage. (b) Direct captioning with large language models (LLMs) generates descriptions in a single forward pass, but often suffers from hallucinations or omissions—particularly in capturing fine-grained visual details such as clothing, accessories, or scene context. (c) Our proposed multi-turn strategy simulates an interactive dialogue with the MLLM, progressively enriching descriptions through targeted Q&A, yielding more expressive, accurate, and human-aligned captions.
• Figure 2: Overview of the proposed CTGI framework for text-based person search. The framework consists of two stages: (1) Training-time generation: MTG simulates multi-turn dialogue to iteratively enrich captions, while a reconstructor synthesizes pseudo-labels using structured prompts; and (2) Inference-time retrieval: MTI refines user queries through MLLM-driven Q&A, enhancing alignment between the query and candidate images for improved re-ranking.
• Figure 3: Top-10 retrieval results on the RSTPReid dataset. The first column is the ground-truth image. The first row shows retrieval results using IRRA; the second row shows results after applying IRRA with MTI. Refined queries generated by multi-turn interaction are shown alongside each example. Green borders indicate correct matches.