Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMs

TL;DR

This work tackles the hallucination problem in multimodal LLMs by identifying that relying on a single deep visual layer limits grounding. It proposes TGIF, a text-guided inter-layer fusion module that treats vision encoder layers as depth-wise experts and performs prompt-conditioned, external fusion without updating the vision encoder. Through a two-stage training setup on LLaVA-1.5 with CLIP-ViT-L/14 and Vicuna-7B, TGIF achieves state-of-the-art 7B results on hallucination benchmarks and substantial gains on OCR tasks, while preserving or enhancing general reasoning. The results demonstrate that dynamic, hierarchy-aware fusion improves visual grounding and trustworthiness in MLLMs with minimal computational overhead and no vision-side fine-tuning.

Abstract

Multimodal large language models (MLLMs) typically rely on a single late-layer feature from a frozen vision encoder, leaving the encoder's rich hierarchy of visual cues under-utilized. MLLMs still suffer from visually ungrounded hallucinations, often relying on language priors rather than image evidence. While many prior mitigation strategies operate on the text side, they leave the visual representation unchanged and do not exploit the rich hierarchy of features encoded across vision layers. Existing multi-layer fusion methods partially address this limitation but remain static, applying the same layer mixture regardless of the query. In this work, we introduce TGIF (Text-Guided Inter-layer Fusion), a lightweight module that treats encoder layers as depth-wise "experts" and predicts a prompt-dependent fusion of visual features. TGIF follows the principle of direct external fusion, requires no vision-encoder updates, and adds minimal overhead. Integrated into LLaVA-1.5-7B, TGIF provides consistent improvements across hallucination, OCR, and VQA benchmarks, while preserving or improving performance on ScienceQA, GQA, and MMBench. These results suggest that query-conditioned, hierarchy-aware fusion is an effective way to strengthen visual grounding and reduce hallucination in modern MLLMs.

Figures (3)

• Figure 2: Overview of the proposed Text-Guided Inter-Layer Fusion (TGIF) framework. TGIF dynamically integrates hierarchical visual features from a frozen vision encoder based on the textual query. The image is first processed by the Vision Transformer (ViT), producing multi-layer representations $\{F_l\}$ that capture progressively abstract semantics. The Layer Router receives the text embedding $f_{\text{text}}$ and outputs a soft distribution over encoder layers $w_{\text{layer}}$ through an MLP and softmax. These weights determine the contribution of each layer to the fused visual feature $F_{\text{fused}}$, which is then projected to the text space by a lightweight MLP connector. The fused multimodal tokens are concatenated with the tokenized text and fed into the LLM for reasoning and response generation.
• Figure 3: Router layer selection patterns across different question categories. This heatmap visualizes the router's learned weights for selecting vision transformer (ViT) layers across three categories of questions: General, Hallucination Detection, and OCR/Detail Recognition. Each row corresponds to one question, and each column indicates a specific ViT layer. Brighter colors denote higher selection weight for that layer.
• Figure 4: Effect of load balancing on the VQA–hallucination trade-off. Each point shows the average VQA score (ScienceQA, GQA, TextVQA) versus POPE accuracy for a different router / load-balancing configuration. We annotate all load-balancing settings next to their corresponding points.