Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FreeVA: Offline MLLM as Training-Free Video Assistant

Wenhao Wu

TL;DR

This paper investigates a training-free approach, FreeVA, to extend image-based Multimodal Large Language Models (MLLMs) to video understanding. By sampling video frames and applying parameter-free temporal aggregation to the image MLLM tokens, FreeVA achieves strong zero-shot video QA performance, sometimes surpassing models trained with video instruction tuning. It also reveals that video instruction tuning datasets like VideoInstruct-100K can hurt performance and that GPT-3.5 API version changes markedly affect evaluation results. The work provides a simple, plug-and-play baseline to standardize evaluation of video MLLMs and encourages rethinking whether current video MLLMs truly gain knowledge beyond image MLLMs. The code is available at https://github.com/whwu95/FreeVA.

Abstract

This paper undertakes an empirical study to revisit the latest advancements in Multimodal Large Language Models (MLLMs): Video Assistant. This study, namely FreeVA, aims to extend existing image-based MLLM to the video domain in a training-free manner. The study provides an essential, yet must-know baseline, and reveals several surprising findings: 1) FreeVA, leveraging only offline image-based MLLM without additional training, excels in zero-shot video question-answering (e.g., MSVD-QA, ActivityNet-QA, and MSRVTT-QA), even surpassing state-of-the-art methods that involve video instruction tuning. 2) While mainstream video-based MLLMs typically initialize with an image-based MLLM (e.g., LLaVA) and then fine-tune using video instruction tuning, the study indicates that utilizing the widely adopted VideoInstruct-100K for video instruction tuning doesn't actually lead to better performance compared to not training at all. 3) The commonly used evaluation metrics in existing works are significantly influenced by changes in the GPT API version over time. If ignored, this could affect the fairness and uniformity of comparisons between different methods and impact the analysis and judgment of researchers in the field. The advancement of MLLMs is currently thriving, drawing numerous researchers into the field. We aim for this work to serve as a plug-and-play, simple yet effective baseline, encouraging the direct evaluation of existing MLLMs in video domain while also standardizing the field of video conversational models to a certain extent. Also, we encourage researchers to reconsider: Have current video MLLM methods truly acquired knowledge beyond image MLLM? Code is available at https://github.com/whwu95/FreeVA

FreeVA: Offline MLLM as Training-Free Video Assistant

TL;DR

This paper investigates a training-free approach, FreeVA, to extend image-based Multimodal Large Language Models (MLLMs) to video understanding. By sampling video frames and applying parameter-free temporal aggregation to the image MLLM tokens, FreeVA achieves strong zero-shot video QA performance, sometimes surpassing models trained with video instruction tuning. It also reveals that video instruction tuning datasets like VideoInstruct-100K can hurt performance and that GPT-3.5 API version changes markedly affect evaluation results. The work provides a simple, plug-and-play baseline to standardize evaluation of video MLLMs and encourages rethinking whether current video MLLMs truly gain knowledge beyond image MLLMs. The code is available at https://github.com/whwu95/FreeVA.

Abstract

This paper undertakes an empirical study to revisit the latest advancements in Multimodal Large Language Models (MLLMs): Video Assistant. This study, namely FreeVA, aims to extend existing image-based MLLM to the video domain in a training-free manner. The study provides an essential, yet must-know baseline, and reveals several surprising findings: 1) FreeVA, leveraging only offline image-based MLLM without additional training, excels in zero-shot video question-answering (e.g., MSVD-QA, ActivityNet-QA, and MSRVTT-QA), even surpassing state-of-the-art methods that involve video instruction tuning. 2) While mainstream video-based MLLMs typically initialize with an image-based MLLM (e.g., LLaVA) and then fine-tune using video instruction tuning, the study indicates that utilizing the widely adopted VideoInstruct-100K for video instruction tuning doesn't actually lead to better performance compared to not training at all. 3) The commonly used evaluation metrics in existing works are significantly influenced by changes in the GPT API version over time. If ignored, this could affect the fairness and uniformity of comparisons between different methods and impact the analysis and judgment of researchers in the field. The advancement of MLLMs is currently thriving, drawing numerous researchers into the field. We aim for this work to serve as a plug-and-play, simple yet effective baseline, encouraging the direct evaluation of existing MLLMs in video domain while also standardizing the field of video conversational models to a certain extent. Also, we encourage researchers to reconsider: Have current video MLLM methods truly acquired knowledge beyond image MLLM? Code is available at https://github.com/whwu95/FreeVA
Paper Structure (11 sections, 2 figures, 8 tables)

This paper contains 11 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Preliminaries: MLLM Architecture Overview
  5. FreeVA: Training-Free Video Assistant
  6. Experiments
  7. Experimental Setup
  8. Empirical Observations on Basic Factors
  9. Main Results
  10. Limitation and Discussion
  11. Conclusion

Figures (2)

  • Figure 1: An illustration of (a) an overview of the image MLLM inference process and (b) our proposed FreeVA for zero-shot video inference using existing image MLLMs.
  • Figure 2: Examples of FreeVA's capabilities in multimodal video understanding, illustrating its proficiency in generating responses based on provided instructions. Please zoom in for best view.