Not Every Patch is Needed: Towards a More Efficient and Effective Backbone for Video-based Person Re-identification

TL;DR

The paper addresses the high computational cost of video-based person ReID by proposing a selective patch-based backbone. It introduces a patch selection mechanism guided by selection-determinate features and a patch-sparse transformer (PSFormer) that injects pseudo global context to compensate for sparse inputs, with a GOP-based temporal structure and dynamic routing between patch-wise and global context generation. The method achieves substantial computational savings (up to 74% fewer MACs than ViT-B and 28% less than ResNet50) while maintaining or surpassing accuracy of strong backbones, and demonstrates strong plug-and-play performance when used as a backbone for existing ReID models. This work offers a practical, scalable approach for real-time video ReID systems by balancing selective feature extraction with effective global context through dynamic, hardware-friendly design.

Abstract

This paper proposes a new effective and efficient plug-and-play backbone for video-based person re-identification (ReID). Conventional video-based ReID methods typically use CNN or transformer backbones to extract deep features for every position in every sampled video frame. Here, we argue that this exhaustive feature extraction could be unnecessary, since we find that different frames in a ReID video often exhibit small differences and contain many similar regions due to the relatively slight movements of human beings. Inspired by this, a more selective, efficient paradigm is explored in this paper. Specifically, we introduce a patch selection mechanism to reduce computational cost by choosing only the crucial and non-repetitive patches for feature extraction. Additionally, we present a novel network structure that generates and utilizes pseudo frame global context to address the issue of incomplete views resulting from sparse inputs. By incorporating these new designs, our backbone can achieve both high performance and low computational cost. Extensive experiments on multiple datasets show that our approach reduces the computational cost by 74\% compared to ViT-B and 28\% compared to ResNet50, while the accuracy is on par with ViT-B and outperforms ResNet50 significantly.

Figures (6)

• Figure 1: Overview of our framework, including two components: (a) Patch Selection and (b) Patch-sparse Transformer. Note that, for simplicity of illustration, this figure only presents one I-frame with one P-frame in the patch-sparse transformer.
• Figure 2: Visualization of selected patches for the 1st, 3rd, 5th, and 7th frames in a video clip. The selected patches are marked by yellow masks. (Best viewed in color)
• Figure 3: Convergence curves on the MARS dataset.
• Figure 4: Visualization of the eigenvector $y_{1}$ associated with the smallest nonzero eigenvalue obtained through the spectral decomposition method illustrated in Sec.\ref{['text_spectral']}
• Figure 5: Ablation study for the weight of error-constraint loss $\mathcal{L}_{error}$ on MARS dataset.