LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding

TL;DR

Video understanding requires modeling temporal dynamics and semantic context beyond static images. LLaVA-Octopus introduces an instruction-driven adaptive projector fusion to leverage multiple visual projectors and enhance multimodal video understanding. The approach uses an instruction-driven router to dynamically fuse image-based, spatial-temporal, and token-compress projectors, trained via a two-stage process of multi-task pre-training and instruction tuning. Across MVBench, VideoMME, and long-video benchmarks, the method achieves strong or state-of-the-art performance, demonstrating robust cross-task generalization and the practical potential of adaptive projector fusion for video understanding.

Abstract

In this paper, we introduce LLaVA-Octopus, a novel video multimodal large language model. LLaVA-Octopus adaptively weights features from different visual projectors based on user instructions, enabling us to leverage the complementary strengths of each projector. We observe that different visual projectors exhibit distinct characteristics when handling specific tasks. For instance, some projectors excel at capturing static details, while others are more effective at processing temporal information, and some are better suited for tasks requiring temporal coherence. By dynamically adjusting feature weights according to user instructions, LLaVA-Octopus dynamically selects and combines the most suitable features, significantly enhancing the model's performance in multimodal tasks. Experimental results demonstrate that LLaVA-Octopus achieves excellent performance across multiple benchmarks, especially in tasks such as video question answering, long video understanding, and comprehensive multi-choices benchmarks, highlighting its broad application potential.

Figures (10)

• Figure 1: Comparison of Different MLLM Paradigms. In the classical paradigm, user instructions are fed into the LLM solely as text tokens. While the instruction-involved paradigm facilitates interaction between instructions and visual features, it is constrained by a single projector. Our proposed instruction-driven projector fusion paradigm designs a projector fusion router, which dynamically adjusts the weights of different types of visual projectors based on user instructions to produce the fused visual tokens.
• Figure 2: Comparisons of three representative methods under different video understanding scenarios. VideoChat2-HD li2024mvbenchcomprehensivemultimodalvideo uses image-based projector while VideoLLaMa2 damonlpsg2024videollama2 and LLaMA-VID li2024llamavid use spatial-temporal projector and token-compress projector, respectively. The results indicate that different visual projectors perform well in their appropriate domains while exhibiting poorer performance in other scenarios. More examples will be provided in the supplementary materials.
• Figure 3: Pipeline of the proposed LLaVA-Octopus model. Our LLaVA-Octopus proposes an instruction-driven adaptive projector that involves three types of visual projectors to enhance the model's ability in multimodal tasks.
• Figure 4: Multimodal Data Distribution and Data Format.$<$image$>$ and $<$video$>$ represent visual tokens from image and video data, respectively.
• Figure 5: Qualitative Results of LLaVA-Octopus. Compared to using a single type of projector, LLaVA-Octopus is capable of leveraging the strengths of different projectors, thereby transcending the limited advantages of a single projector. This enables LLaVA-Octopus to achieve excellent performance across various tasks.