Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices

TL;DR

The paper tackles how to best leverage multi-layer visual features in multimodal LLMs by jointly studying visual layer selection and fusion strategies. It introduces two criteria for layer selection (similarity-based grouping and proportion-based division) and compares internal vs external fusion across modular and direct patterns using a lightweight Mini-LLaVA baseline. Key findings show that selecting representative features from the beginning and middle stages, combined with including ending-stage tokens, yields strong generalization, while external direct fusion consistently delivers the strongest performance and stability across settings. The work also demonstrates that larger training data and model components can reduce gaps for internal fusion, and it provides practical guidance and public code for reproducing and extending these insights.

Abstract

Multimodal Large Language Models (MLLMs) have made significant advancements in recent years, with visual features playing an increasingly critical role in enhancing model performance. However, the integration of multi-layer visual features in MLLMs remains underexplored, particularly with regard to optimal layer selection and fusion strategies. Existing methods often rely on arbitrary design choices, leading to suboptimal outcomes. In this paper, we systematically investigate two core aspects of multi-layer visual feature fusion: (1) selecting the most effective visual layers and (2) identifying the best fusion approach with the language model. Our experiments reveal that while combining visual features from multiple stages improves generalization, incorporating additional features from the same stage typically leads to diminished performance. Furthermore, we find that direct fusion of multi-layer visual features at the input stage consistently yields superior and more stable performance across various configurations. We make all our code publicly available: https://github.com/EIT-NLP/Layer_Select_Fuse_for_MLLM.

Figures (8)

• Figure 1: Different Visual Features and Fusion Paradigms. (a) and (b) illustrate the acquisition methods for single-layer and multi-layer visual features, respectively. (c), (d), (e), and (f) display four different fusion strategies: the first two categorize fusion strategies based on fusion position, while the latter two classify fusion strategies based on fusion pattern.
• Figure 2: Comparison of Similarity-Based and Proportion-Based Visual Layer Selection.
• Figure 3: Framework of the four fusion strategies studied in this work. Blue lines represent the path passing through the projector.
• Figure 4: Pre-cross attention loss curves in pre-training stage under different layer sets.
• Figure 5: The performance trend of different fusion strategies adopting the Triple visual layers as the training dataset increases. The abbreviations represent different fusion configurations: E denotes External fusion, I denotes Internal fusion, D stands for Direct fusion, and M indicates Modular fusion.