Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation

Jingnan Luo, Mingqi Gao, Jun Liu, Bin-Bin Gao, Feng Zheng

Abstract

The prosperity of Multimodal Large Language Models (MLLMs) has stimulated the demand for video reasoning segmentation, which aims to segment video objects based on human instructions. Previous studies rely on unidirectional and implicit text-trajectory alignment, which struggles with trajectory perception when faced with severe video dynamics. In this work, we propose TrajSeg, a simple and unified framework built upon MLLMs. Concretely, we introduce bidirectional text-trajectory alignment, where MLLMs accept grounding-intended (text-to-trajectory) and captioning-intended (trajectory-to-text) instructions. This way, MLLMs can benefit from enhanced correspondence and better perceive object trajectories in videos. The mask generation from trajectories is achieved via a frame-level content integration (FCI) module and a unified mask decoder. The former adapts the MLLM-parsed trajectory-level token to frame-specific information. The latter unifies segmentation for all frames into a single structure, enabling the proposed framework to be simplified and end-to-end trainable. Extensive experiments on referring and reasoning video segmentation datasets demonstrate the effectiveness of TrajSeg, which outperforms all video reasoning segmentation methods on all metrics. The code will be publicly available at https://github.com/haodi19/TrajSeg.

Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation

Abstract

The prosperity of Multimodal Large Language Models (MLLMs) has stimulated the demand for video reasoning segmentation, which aims to segment video objects based on human instructions. Previous studies rely on unidirectional and implicit text-trajectory alignment, which struggles with trajectory perception when faced with severe video dynamics. In this work, we propose TrajSeg, a simple and unified framework built upon MLLMs. Concretely, we introduce bidirectional text-trajectory alignment, where MLLMs accept grounding-intended (text-to-trajectory) and captioning-intended (trajectory-to-text) instructions. This way, MLLMs can benefit from enhanced correspondence and better perceive object trajectories in videos. The mask generation from trajectories is achieved via a frame-level content integration (FCI) module and a unified mask decoder. The former adapts the MLLM-parsed trajectory-level token to frame-specific information. The latter unifies segmentation for all frames into a single structure, enabling the proposed framework to be simplified and end-to-end trainable. Extensive experiments on referring and reasoning video segmentation datasets demonstrate the effectiveness of TrajSeg, which outperforms all video reasoning segmentation methods on all metrics. The code will be publicly available at https://github.com/haodi19/TrajSeg.
Paper Structure (26 sections, 6 equations, 8 figures, 5 tables)

This paper contains 26 sections, 6 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Referring Video Object Segmentation (RVOS)
  4. Multimodal Large Language Model (MLLM)
  5. Method
  6. Overview
  7. Bidirectional Text-Trajectory Alignment
  8. Frame-level Content Integration Module
  9. Unified Mask Generator
  10. Training Objectives
  11. Experiments
  12. Experimental Setting
  13. Training Data
  14. Implementation Details
  15. Evaluation Protocols
  16. ...and 11 more sections

Figures (8)

  • Figure 1: Comparison of existing diagram yan2024visabai2024one and ours. (a) learns MLLM via unidirectional alignment ("text-to-trajectory"). Ours considers "text-to-trajectory" and "trajectory-to-text" to enhance their correspondence ([regular] is the placeholder for trajectory features). Moreover, (b) uses a frame-content integration (FCI) module to refine trajectory tokenization with frame-specific clues. For mask generation, (a) segments key and non-key frames with separately optimized models. (b) supports flexible inputs and unifies all frame segmentation in a single structure, enabling a simplified and end-to-end trainable framework.
  • Figure 2: Diagram of TrajSeg. (a) Overall framework; (b) Detailed structure of the unified mask generator; (c) Illustration of how the mask generator processes continuous key & non-key frames.
  • Figure 3: Comparisons of TrajSeg and VISA-7B yan2024visa on ReVOS. Blue boxes are Ground Truth objects.
  • Figure 4: Visualization of attention in the MLLM and corresponding masks. (a) masks w/ Bi-Align. (b) attention w/ Bi-Align. (c) masks w/o Bi-Align. (d) attention w/o Bi-Align.
  • Figure 5: Visualization of captioning-intended instructions.
  • ...and 3 more figures