Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VILLS -- Video-Image Learning to Learn Semantics for Person Re-Identification

Siyuan Huang, Ram Prabhakar, Yuxiang Guo, Rama Chellappa, Cheng Peng

TL;DR

VILLS (Video-Image Learning to Learn Semantics), a self-supervised method that jointly learns spatial and temporal features from images and videos, establishes a new State-of-The-Art that significantly outperforms existing image and video-based methods.

Abstract

Person Re-identification is a research area with significant real world applications. Despite recent progress, existing methods face challenges in robust re-identification in the wild, e.g., by focusing only on a particular modality and on unreliable patterns such as clothing. A generalized method is highly desired, but remains elusive to achieve due to issues such as the trade-off between spatial and temporal resolution and imperfect feature extraction. We propose VILLS (Video-Image Learning to Learn Semantics), a self-supervised method that jointly learns spatial and temporal features from images and videos. VILLS first designs a local semantic extraction module that adaptively extracts semantically consistent and robust spatial features. Then, VILLS designs a unified feature learning and adaptation module to represent image and video modalities in a consistent feature space. By Leveraging self-supervised, large-scale pre-training, VILLS establishes a new State-of-The-Art that significantly outperforms existing image and video-based methods.

VILLS -- Video-Image Learning to Learn Semantics for Person Re-Identification

TL;DR

VILLS (Video-Image Learning to Learn Semantics), a self-supervised method that jointly learns spatial and temporal features from images and videos, establishes a new State-of-The-Art that significantly outperforms existing image and video-based methods.

Abstract

Person Re-identification is a research area with significant real world applications. Despite recent progress, existing methods face challenges in robust re-identification in the wild, e.g., by focusing only on a particular modality and on unreliable patterns such as clothing. A generalized method is highly desired, but remains elusive to achieve due to issues such as the trade-off between spatial and temporal resolution and imperfect feature extraction. We propose VILLS (Video-Image Learning to Learn Semantics), a self-supervised method that jointly learns spatial and temporal features from images and videos. VILLS first designs a local semantic extraction module that adaptively extracts semantically consistent and robust spatial features. Then, VILLS designs a unified feature learning and adaptation module to represent image and video modalities in a consistent feature space. By Leveraging self-supervised, large-scale pre-training, VILLS establishes a new State-of-The-Art that significantly outperforms existing image and video-based methods.
Paper Structure (16 sections, 14 equations, 3 figures, 3 tables)

This paper contains 16 sections, 14 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Local Semantic Extraction Module
  5. Unified Feature Learning and Adaptation Module
  6. Experiments
  7. Datasets and Evaluation Metrics
  8. Main Results
  9. Image-based ReID
  10. Video-based ReID
  11. Image-Video Mix ReID
  12. Ablation Studies
  13. Significance of Components
  14. Losses and Balancing Parameters
  15. Conclusion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: VILLS jointly learns spatial and temporal features from both images and videos, achieving state-of-the-art performance across image and video ReID tasks.
  • Figure 2: The pipeline of proposed VILLS. We introduce a Local Semantic Extraction module that can adaptively extract semantically consistent features from any area. Then, we introduce a Unified Feature Learning and Adaptation module to uniformly represent and jointly learn image and video semantics.
  • Figure 3: Visualization of attentions. VILLS demonstrates semantically consistent attention patterns. Please refer to the Supplemental Material for more details.