Breaking the Encoder Barrier for Seamless Video-Language Understanding

TL;DR

ELVA presents a fully encoder-free Video-LLM that learns spatiotemporal representations directly inside the LLM by integrating native video tokenization, a lightweight video patch embedding layer, and a bottom-up hierarchical token merging strategy. A video guidance supervisor, combining tube-wise alignment and frame-wise contrastive losses with a generative objective, directs learning from raw pixels, while a hybrid-resolution inference mechanism balances fidelity and efficiency for long videos. Trained on 7M video-text pairs across a three-stage curriculum including recaption pretraining and stage-wise pretraining plus SFT, ELVA achieves competitive performance with encoder-based models but reduces FLOPs by up to $95\%$ and latency by $92\%$, enabling scalable, real-time video understanding. The work demonstrates that encoder-free Video-LLMs can reach state-of-the-art-like capabilities while offering substantial practical benefits in compute and latency, informing future multimodal model design and deployment.

Abstract

Most Video-Large Language Models (Video-LLMs) adopt an encoder-decoder framework, where a vision encoder extracts frame-wise features for processing by a language model. However, this approach incurs high computational costs, introduces resolution biases, and struggles to capture fine-grained multimodal interactions. To overcome these limitations, we propose ELVA, an encoder-free Video-LLM that directly models nuanced video-language interactions without relying on a vision encoder. ELVA employs token merging to construct a bottom-up hierarchical representation and incorporates a video guidance supervisor for direct spatiotemporal representation learning. Additionally, a hybrid-resolution mechanism strategically integrates high- and low-resolution frames as inputs to achieve an optimal balance between performance and efficiency. With only 7M publicly available video-text pairs, ELVA achieves performance on par with encoder-based Video-LLMs while reducing FLOPs by up to 95\% and inference latency by 92\%, offering a scalable and efficient solution for real-time video understanding.

Figures (4)

• Figure 1: Encoder-based vs. encoder-free Video-LLMs. Encoder-based models connect a vision encoder with an LLM, encountering severe issues in effectively handling video data. In contrast, ELVA adopts an encoder-free architecture, directly integrating video perception and language modeling within a unified framework, achieving dramatically higher efficiency.
• Figure 2: Framework illustration of ELVA. ELVA is an encoder-free Video-LLM that captures nuanced video-language interactions through a bottom-up hierarchical design with layer-wise token merging. A video guidance loss enhances spatiotemporal representation learning, while a hybrid-resolution inference strategy optimizes the balance between computational efficiency and content fidelity.
• Figure 3: The effect of hierarchical merging on accuracy and inference speedup across layer compression ratios. VideoMME (Acc) is presented using a line chart, while the speedup ratio is illustrated with a bar chart.
• Figure 4: Effect of pre-training data size across two pre-training stages. All checkpoint results are reported after undergoing the same instruction tuning process.