Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MLLMReID: Multimodal Large Language Model-based Person Re-identification

Shan Yang, Yongfei Zhang

TL;DR

This work tackles person re-identification with Multimodal Large Language Models (MLLMs) by addressing instruction overfitting and non-synchronous visual encoder training. It introduces MLLMReID, featuring a Common Instruction that uses simple continuation prompts and a SyncReID module that directly optimizes the visual encoder with ReID losses via LLМ-derived latent features. The method achieves measurable gains on MSMT17 and competitive results on other datasets, and demonstrates improved generalization in cross-dataset settings. The approach suggests a practical path to leverage MLLMs for broader multimodal tasks beyond ReID, by unifying instruction design with task-aligned training.

Abstract

Multimodal large language models (MLLM) have achieved satisfactory results in many tasks. However, their performance in the task of ReID (ReID) has not been explored to date. This paper will investigate how to adapt them for the task of ReID. An intuitive idea is to fine-tune MLLM with ReID image-text datasets, and then use their visual encoder as a backbone for ReID. However, there still exist two apparent issues: (1) Designing instructions for ReID, MLLMs may overfit specific instructions, and designing a variety of instructions will lead to higher costs. (2) When fine-tuning the visual encoder of a MLLM, it is not trained synchronously with the ReID task. As a result, the effectiveness of the visual encoder fine-tuning cannot be directly reflected in the performance of the ReID task. To address these problems, this paper proposes MLLMReID: Multimodal Large Language Model-based ReID. Firstly, we proposed Common Instruction, a simple approach that leverages the essence ability of LLMs to continue writing, avoiding complex and diverse instruction design. Secondly, we propose a multi-task learning-based synchronization module to ensure that the visual encoder of the MLLM is trained synchronously with the ReID task. The experimental results demonstrate the superiority of our method.

MLLMReID: Multimodal Large Language Model-based Person Re-identification

TL;DR

This work tackles person re-identification with Multimodal Large Language Models (MLLMs) by addressing instruction overfitting and non-synchronous visual encoder training. It introduces MLLMReID, featuring a Common Instruction that uses simple continuation prompts and a SyncReID module that directly optimizes the visual encoder with ReID losses via LLМ-derived latent features. The method achieves measurable gains on MSMT17 and competitive results on other datasets, and demonstrates improved generalization in cross-dataset settings. The approach suggests a practical path to leverage MLLMs for broader multimodal tasks beyond ReID, by unifying instruction design with task-aligned training.

Abstract

Multimodal large language models (MLLM) have achieved satisfactory results in many tasks. However, their performance in the task of ReID (ReID) has not been explored to date. This paper will investigate how to adapt them for the task of ReID. An intuitive idea is to fine-tune MLLM with ReID image-text datasets, and then use their visual encoder as a backbone for ReID. However, there still exist two apparent issues: (1) Designing instructions for ReID, MLLMs may overfit specific instructions, and designing a variety of instructions will lead to higher costs. (2) When fine-tuning the visual encoder of a MLLM, it is not trained synchronously with the ReID task. As a result, the effectiveness of the visual encoder fine-tuning cannot be directly reflected in the performance of the ReID task. To address these problems, this paper proposes MLLMReID: Multimodal Large Language Model-based ReID. Firstly, we proposed Common Instruction, a simple approach that leverages the essence ability of LLMs to continue writing, avoiding complex and diverse instruction design. Secondly, we propose a multi-task learning-based synchronization module to ensure that the visual encoder of the MLLM is trained synchronously with the ReID task. The experimental results demonstrate the superiority of our method.
Paper Structure (16 sections, 4 equations, 3 figures, 5 tables)

This paper contains 16 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image-Text Multimodal person re-identification
  4. Method
  5. Architecture of MLLM
  6. Common Instruction
  7. SyncReID
  8. Overall Loss
  9. Experiments
  10. Experiment Setting Overview
  11. Performance Comparison
  12. Analyzing and Comparing Performance Metrics
  13. Performance Comparison in Generalizable ReID Task
  14. Ablation Study
  15. Case Study
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Naive Implementation of Multimodal Large Language Model for Person Re-identification.
  • Figure 2: Overview of our proposed approach: Multimodal Large Language Model-based Person Re-identification.
  • Figure 3: Schematic representation of ReID retrieval results. The first and second rows represent the retrieval results of the baseline system and our method, respectively.