Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DRFormer: A Dual-Regularized Bidirectional Transformer for Person Re-identification

Ying Shu, Pujian Zhan, Huiqi Yang, Hehe Fan, Youfang Lin, Kai Lv

TL;DR

DRFormer addresses the challenge of reconciling fine-grained local details with global semantic context for person ReID. It introduces a dual-regularized bidirectional transformer that fuses DINO and CLIP representations, using an intra-model token diversity regularizer to encourage diverse, complementary features and an inter-model bias regularizer to balance contributions from the two models. The framework achieves competitive or state-of-the-art results across five benchmarks, with ablations confirming the benefits of each regularizer and the bidirectional fusion design. This approach demonstrates the value of integrating vision foundation and vision-language model strengths to improve reliability under occlusion and pose variation.

Abstract

Both fine-grained discriminative details and global semantic features can contribute to solving person re-identification challenges, such as occlusion and pose variations. Vision foundation models (\textit{e.g.}, DINO) excel at mining local textures, and vision-language models (\textit{e.g.}, CLIP) capture strong global semantic difference. Existing methods predominantly rely on a single paradigm, neglecting the potential benefits of their integration. In this paper, we analyze the complementary roles of these two architectures and propose a framework to synergize their strengths by a \textbf{D}ual-\textbf{R}egularized Bidirectional \textbf{Transformer} (\textbf{DRFormer}). The dual-regularization mechanism ensures diverse feature extraction and achieves a better balance in the contributions of the two models. Extensive experiments on five benchmarks show that our method effectively harmonizes local and global representations, achieving competitive performance against state-of-the-art methods.

DRFormer: A Dual-Regularized Bidirectional Transformer for Person Re-identification

TL;DR

DRFormer addresses the challenge of reconciling fine-grained local details with global semantic context for person ReID. It introduces a dual-regularized bidirectional transformer that fuses DINO and CLIP representations, using an intra-model token diversity regularizer to encourage diverse, complementary features and an inter-model bias regularizer to balance contributions from the two models. The framework achieves competitive or state-of-the-art results across five benchmarks, with ablations confirming the benefits of each regularizer and the bidirectional fusion design. This approach demonstrates the value of integrating vision foundation and vision-language model strengths to improve reliability under occlusion and pose variation.

Abstract

Both fine-grained discriminative details and global semantic features can contribute to solving person re-identification challenges, such as occlusion and pose variations. Vision foundation models (\textit{e.g.}, DINO) excel at mining local textures, and vision-language models (\textit{e.g.}, CLIP) capture strong global semantic difference. Existing methods predominantly rely on a single paradigm, neglecting the potential benefits of their integration. In this paper, we analyze the complementary roles of these two architectures and propose a framework to synergize their strengths by a \textbf{D}ual-\textbf{R}egularized Bidirectional \textbf{Transformer} (\textbf{DRFormer}). The dual-regularization mechanism ensures diverse feature extraction and achieves a better balance in the contributions of the two models. Extensive experiments on five benchmarks show that our method effectively harmonizes local and global representations, achieving competitive performance against state-of-the-art methods.
Paper Structure (21 sections, 1 theorem, 17 equations, 6 figures, 7 tables)

This paper contains 21 sections, 1 theorem, 17 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Person Re-identification
  4. Large-scale Pre-training Models
  5. Feature Fusion Methods
  6. Proposed Method
  7. Key Observations and Motivation
  8. Bidirectional Fusion Transformer
  9. The Intra-model Token Diversity Regularizer
  10. The Inter-model Bias Regularizer
  11. Training Loss
  12. Experiments
  13. Experimental Setup
  14. Comparison with State-of-the-Art Methods
  15. Ablation Study
  16. ...and 6 more sections

Key Result

Proposition 3.1

Given a weighted fusion $f(s) = w_0s^D +w_1 s^C$ with $w_0, w_1 > 0$ and $w_0 + w_1 = 1$, minimizing the generalization error requires minimizing the prediction bias of individual branches.

Figures (6)

  • Figure 1: Comparison of attention patterns and feature distributions for DINO, CLIP and our method. The left column visualizes attention weights for all tokens and a selected token: Figure (a) shows the query image token at the red dot, while Figures (c) and (e) depict attention weights of the [CLS] token. The right column presents feature distributions of fine-tuned DINO, fine-tuned CLIP, and our method on some samples.
  • Figure 2: The left and right columns illustrate the motivation and effects of the intra-model and inter-model regularizers, respectively. Left: Figures (a) and (b) depict the attention weights of the two learnable tokens before and after the intra-model regularizer on DukeMTMC dataset. Figure (c) shows the performance gain (2.5% $\uparrow$). Right: Figures (d) and (e) present the DINO and CLIP accuracy curves before and after adding the inter-model regularizer on Market-1501 dataset. Figure (f) shows the performance improvement (1.8% $\uparrow$).
  • Figure 3: The framework of our method. DRFormer mainly consists of a CLIP image encoder, a DINO encoder, and a bidirectional fusion transformer. An input image is split into patches, which, along with learnable tokens, are fed separately into CLIP and DINO. The outputs of the learnable tokens from the two models serve two purposes: performing intra-model regularization and acting as queries for cross-attention. The two outputs of the fusion module are concatenated and fed into the linear classifier. Additionally, each output is also individually passed to the classifier for inter-model regularization.
  • Figure 4: Visualization of the experiment results with different numbers of learnable tokens on Market-1501 and DukeMTMC datasets.
  • Figure 5: Visualization of attention weights on DukeMTMC and Occluded-Duke datasets.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Proposition 3.1