MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model

TL;DR

MaVEn tackles the limitation of current MLLMs in multi-image reasoning by introducing a multi-granularity visual encoding that fuses discrete symbol sequences with continuous feature embeddings. It employs a unified multimodal vocabulary $N_u = N + N_v$, a dynamic reduction mechanism that yields $m_c = n_c \times \alpha$, and a four-stage training paradigm to align the encodings with the LLM. Empirical results on DemonBench and SEED-Bench show strong gains in multi-image reasoning, while VQA/MMBench and MME benchmarks indicate complementary improvements in single-image tasks, all achieved with an efficient token reduction that preserves essential information. By bridging discrete semantic abstractions and continuous details, MaVEn advances robust multimodal reasoning and offers a scalable path for improving MLLMs’ handling of multi-image inputs.

Abstract

This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. Current MLLMs primarily focus on single-image visual understanding, limiting their ability to interpret and integrate information across multiple images. MaVEn addresses this limitation by combining discrete visual symbol sequences, which abstract coarse-grained semantic concepts, with traditional continuous representation sequences that model fine-grained features. This dual approach bridges the semantic gap between visual and textual data, thereby improving the model's ability to process and interpret information from multiple images effectively. Additionally, we design a dynamic reduction mechanism by for long-sequence continuous features to enhance multi-image processing efficiency. Experimental results demonstrate that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.

Figures (6)

• Figure 1: We compared the performance of the classic single-image task trained MLLM LLaVA1.5 LLaVA-1.5 and our model in three multi-image scenarios including Multi Image Reasoning, Knowledge Based VQA and Visual Relation Inference. LLaVA1.5 exhibits significant limitations in multi-image scenarios.
• Figure 2: Subfigure (a) illustrates the structural schematic of our proposed Multi-Granularity Hybrid Encoding, while subfigure (b) demonstrates the mechanism for the reduction of continuous visual tokens under the guidance of discrete visual information.
• Figure 3: The diagram illustrates the training schematic for MaVEn. We divide the training of MaVEn into four stages, where the snowflake icon indicates that the model parameters are frozen during training, and the flame icon indicates that the model parameters are updated during training.
• Figure 4: Evaluation Results of MaVEn on different benchmarks with varying Keeping Ratios.
• Figure 5: This figure visualizes the distribution of discrete tokens in an image containing index 4568 discrete tokens, along with the relevant score computed based on the Patch Selector and the patches chosen according to the relevant score that are most semantically related to the discrete visual tokens.