Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Elysium: Exploring Object-level Perception in Videos via MLLM

Han Wang, Yanjie Wang, Yongjie Ye, Yuxiang Nie, Can Huang

TL;DR

<3-5 sentence high-level summary> Elysium addresses the challenge of enabling object-level perception in videos for multimodal LLMs by introducing ElysiumTrack-1M, a large-scale dataset derived from WebVid-10M with 1.27M trajectories and expressions for SOT, RSOT, and Video-REG. It presents Elysium, an end-to-end trainable MLLM that integrates a visual encoder, a token-compression module (T-Selector), and an LLM to process multiple frames with a controllable token budget. The work includes a two-stage training regime and extensive evaluations across image grounding, VideoQA, SOT, RSOT, and Video-REG, showing state-of-the-art performance and robust object-level reasoning. It also analyzes the impact of token compression and temporal design choices, highlighting the practicality of scalable, plug-free object-level video understanding with MLLMs.

Abstract

Multi-modal Large Language Models (MLLMs) have demonstrated their ability to perceive objects in still images, but their application in video-related tasks, such as object tracking, remains understudied. This lack of exploration is primarily due to two key challenges. Firstly, extensive pretraining on large-scale video datasets is required to equip MLLMs with the capability to perceive objects across multiple frames and understand inter-frame relationships. Secondly, processing a large number of frames within the context window of Large Language Models (LLMs) can impose a significant computational burden. To address the first challenge, we introduce ElysiumTrack-1M, a large-scale video dataset supported for three tasks: Single Object Tracking (SOT), Referring Single Object Tracking (RSOT), and Video Referring Expression Generation (Video-REG). ElysiumTrack-1M contains 1.27 million annotated video frames with corresponding object boxes and descriptions. Leveraging this dataset, we conduct training of MLLMs and propose a token-compression model T-Selector to tackle the second challenge. Our proposed approach, Elysium: Exploring Object-level Perception in Videos via MLLM, is an end-to-end trainable MLLM that attempts to conduct object-level tasks in videos without requiring any additional plug-in or expert models. All codes and datasets are available at https://github.com/Hon-Wong/Elysium.

Elysium: Exploring Object-level Perception in Videos via MLLM

TL;DR

<3-5 sentence high-level summary> Elysium addresses the challenge of enabling object-level perception in videos for multimodal LLMs by introducing ElysiumTrack-1M, a large-scale dataset derived from WebVid-10M with 1.27M trajectories and expressions for SOT, RSOT, and Video-REG. It presents Elysium, an end-to-end trainable MLLM that integrates a visual encoder, a token-compression module (T-Selector), and an LLM to process multiple frames with a controllable token budget. The work includes a two-stage training regime and extensive evaluations across image grounding, VideoQA, SOT, RSOT, and Video-REG, showing state-of-the-art performance and robust object-level reasoning. It also analyzes the impact of token compression and temporal design choices, highlighting the practicality of scalable, plug-free object-level video understanding with MLLMs.

Abstract

Multi-modal Large Language Models (MLLMs) have demonstrated their ability to perceive objects in still images, but their application in video-related tasks, such as object tracking, remains understudied. This lack of exploration is primarily due to two key challenges. Firstly, extensive pretraining on large-scale video datasets is required to equip MLLMs with the capability to perceive objects across multiple frames and understand inter-frame relationships. Secondly, processing a large number of frames within the context window of Large Language Models (LLMs) can impose a significant computational burden. To address the first challenge, we introduce ElysiumTrack-1M, a large-scale video dataset supported for three tasks: Single Object Tracking (SOT), Referring Single Object Tracking (RSOT), and Video Referring Expression Generation (Video-REG). ElysiumTrack-1M contains 1.27 million annotated video frames with corresponding object boxes and descriptions. Leveraging this dataset, we conduct training of MLLMs and propose a token-compression model T-Selector to tackle the second challenge. Our proposed approach, Elysium: Exploring Object-level Perception in Videos via MLLM, is an end-to-end trainable MLLM that attempts to conduct object-level tasks in videos without requiring any additional plug-in or expert models. All codes and datasets are available at https://github.com/Hon-Wong/Elysium.
Paper Structure (17 sections, 2 equations, 4 figures, 7 tables)

This paper contains 17 sections, 2 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. MLLMs
  4. Object-level Tasks in Videos
  5. Construct ElysiumTrack-1M dataset
  6. Introduce RSOT and Video-REG Tasks
  7. Dataset Construction
  8. Evaluations
  9. Elysium
  10. Architecture
  11. Training Setups
  12. Evaluation Setups
  13. Experiments
  14. State-of-the-art Comparisons
  15. Ablation Studies
  16. ...and 2 more sections

Figures (4)

  • Figure 1: The pipeline to construct ElysiumTrack-1M dataset.
  • Figure 2: Samples from ElysiumTrack-1M dataset.
  • Figure 3: The architectures of Elysium. Elysium combines a visual encoder, an LLM, and a T-Selector to connect the visual encoder with the LLM.
  • Figure 4: Visualizations of the Elysium. We showcased the performance of Elysium on several videos. We demonstrated the RSOT capability of Elysium on cases (a) to (e) while showcasing the SOT on case (f) and Video-REG on case (g).