Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text-based Person Search in Full Images via Semantic-Driven Proposal Generation

Shizhou Zhang, De Cheng, Wenlong Luo, Yinghui Xing, Duo Long, Hao Li, Kai Niu, Guoqiang Liang, Yanning Zhang

TL;DR

This work tackles text-based person search in full images, addressing scenarios where bounding boxes and query images are unavailable. It introduces an end-to-end framework that jointly optimizes detection, identification, and image–text embedding, and leverages a Semantic-Driven Region Proposal Network (SDRPN) to focus proposals on text-described candidates. A cross-scale visual–semantic embedding module combines multi-scale visual features with multi-level text features (sentence, sub-sentence, word) to compute robust cross-modal similarities, reinforced by a composite loss that includes detection, identification, and cross-modal terms. The approach is validated on two newly annotated benchmarks, CUHK-SYSU-TBPS and PRW-TBPS, achieving state-of-the-art results and demonstrating practical impact for search in full scenes and surveillance applications.

Abstract

Finding target persons in full scene images with a query of text description has important practical applications in intelligent video surveillance.However, different from the real-world scenarios where the bounding boxes are not available, existing text-based person retrieval methods mainly focus on the cross modal matching between the query text descriptions and the gallery of cropped pedestrian images. To close the gap, we study the problem of text-based person search in full images by proposing a new end-to-end learning framework which jointly optimize the pedestrian detection, identification and visual-semantic feature embedding tasks. To take full advantage of the query text, the semantic features are leveraged to instruct the Region Proposal Network to pay more attention to the text-described proposals. Besides, a cross-scale visual-semantic embedding mechanism is utilized to improve the performance. To validate the proposed method, we collect and annotate two large-scale benchmark datasets based on the widely adopted image-based person search datasets CUHK-SYSU and PRW. Comprehensive experiments are conducted on the two datasets and compared with the baseline methods, our method achieves the state-of-the-art performance.

Text-based Person Search in Full Images via Semantic-Driven Proposal Generation

TL;DR

This work tackles text-based person search in full images, addressing scenarios where bounding boxes and query images are unavailable. It introduces an end-to-end framework that jointly optimizes detection, identification, and image–text embedding, and leverages a Semantic-Driven Region Proposal Network (SDRPN) to focus proposals on text-described candidates. A cross-scale visual–semantic embedding module combines multi-scale visual features with multi-level text features (sentence, sub-sentence, word) to compute robust cross-modal similarities, reinforced by a composite loss that includes detection, identification, and cross-modal terms. The approach is validated on two newly annotated benchmarks, CUHK-SYSU-TBPS and PRW-TBPS, achieving state-of-the-art results and demonstrating practical impact for search in full scenes and surveillance applications.

Abstract

Finding target persons in full scene images with a query of text description has important practical applications in intelligent video surveillance.However, different from the real-world scenarios where the bounding boxes are not available, existing text-based person retrieval methods mainly focus on the cross modal matching between the query text descriptions and the gallery of cropped pedestrian images. To close the gap, we study the problem of text-based person search in full images by proposing a new end-to-end learning framework which jointly optimize the pedestrian detection, identification and visual-semantic feature embedding tasks. To take full advantage of the query text, the semantic features are leveraged to instruct the Region Proposal Network to pay more attention to the text-described proposals. Besides, a cross-scale visual-semantic embedding mechanism is utilized to improve the performance. To validate the proposed method, we collect and annotate two large-scale benchmark datasets based on the widely adopted image-based person search datasets CUHK-SYSU and PRW. Comprehensive experiments are conducted on the two datasets and compared with the baseline methods, our method achieves the state-of-the-art performance.

Paper Structure

This paper contains 18 sections, 7 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Person search
  4. Text-Based Person Retrieval
  5. Variants of Region Proposal Network
  6. Benchmarks for Text-based Person Search
  7. Method
  8. Overview
  9. Semantic-Driven RPN
  10. Proposal-Text Embedding
  11. Overall Loss Function
  12. Inference
  13. Experiment
  14. Datasets and Protocol
  15. Implementation Details
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Comparison of the four tasks. (a) Person ReID. Query: cropped person image. Gallery: cropped person images. (b) Person Search. Query: cropped person image. Gallery: full scene images. (c) Text-based person retrieval. Query: text description. Gallery: cropped person images. (d) Text-based person search. Query: text description. Gallery: full scene images.
  • Figure 2: The word length distributions on the benchmark datasets CUHK-SYSU-TBPS and PRW-TBPS.
  • Figure 3: (a) depicts The overall architecture of the proposed learning framework. (b) shows the process of mixed feature extraction. (c) exhibits the SDRPN.
  • Figure 4: Results comparison with different gallery size of CUHK-SYSU-TBPS
  • Figure 5: Examples of text-based person search.