TVPR: Text-to-Video Person Retrieval and a New Benchmark

TL;DR

This work targets the limitations of text-based retrieval on static images by introducing Text-to-Video Person Retrieval (TVPR) and a new TVPReid benchmark comprising 6559 videos and 13118 natural-language descriptions. It proposes Multielement Feature Guided Fragments Learning (MFGF), which learns text and video fragments and aligns them in two cross-modal spaces through Common Space Learning and Dul-Distilled Space Learning, incorporating a Text Prompter, ViT-based Visual Encoder, and S3D-based Motion Encoder to capture appearance and dynamics. The approach is validated on TVPReid, achieving state-of-the-art results and demonstrating the importance of motion cues and cross-modal distillation for robust text-to-video person matching. The publicly released TVPReid dataset and the proposed framework offer a strong foundation for advancing text-guided video surveillance and multimedia retrieval with improved occlusion handling and motion awareness.

Abstract

Most existing methods for text-based person retrieval focus on text-to-image person retrieval. Nevertheless, due to the lack of dynamic information provided by isolated frames, the performance is hampered when the person is obscured or variable motion details are missed in isolated frames. To overcome this, we propose a novel Text-to-Video Person Retrieval (TVPR) task. Since there is no dataset or benchmark that describes person videos with natural language, we construct a large-scale cross-modal person video dataset containing detailed natural language annotations, termed as Text-to-Video Person Re-identification (TVPReid) dataset. In this paper, we introduce a Multielement Feature Guided Fragments Learning (MFGF) strategy, which leverages the cross-modal text-video representations to provide strong text-visual and text-motion matching information to tackle uncertain occlusion conflicting and variable motion details. Specifically, we establish two potential cross-modal spaces for text and video feature collaborative learning to progressively reduce the semantic difference between text and video. To evaluate the effectiveness of the proposed MFGF, extensive experiments have been conducted on TVPReid dataset. To the best of our knowledge, MFGF is the first successful attempt to use video for text-based person retrieval task and has achieved state-of-the-art performance on TVPReid dataset. The TVPReid dataset will be publicly available to benefit future research.

Figures (8)

• Figure 1: Text-to-video person retrieval can effectively solve the problem of pedestrians being occluded in isolated images, which uses the contextual information provided by the video to make up for the missing features.
• Figure 2: The video contains information about pedestrians' motions and interactions with people and things around them that images cannot provide.
• Figure 3: The overall framework of MFGF. The left is fragments learning of text and video, using powerful text and video understanding networks to learn visual and motion information from text and video fragments. The text and video features extracted from the left are projected into two potential spaces, and then gradually reduce the semantic difference between text and video through visual and motion interactions between multielement features (text feature, video feature, tips, text distilled feature, and video distilled feature)
• Figure 4: ViT and S3D will repeatedly extract some visual and motion features, which will cause redundant information.
• Figure 5: There may be interference factors in the video, which will cause the extracted visual and motion feature to be mixed with impurities that cannot match the text feature.