From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion

TL;DR

Vision-Language Models suffer from an information bottleneck by relying only on final-layer vision features. The paper introduces Cross-Layer Injection (CLI), a lightweight, dynamic many-to-many fusion framework built from Adaptive Multi-Projection (AMP) and Adaptive Gating Fusion (AGF) to enable on-demand access to the full visual hierarchy. By integrating CLI into two VLMs (LLaVA-OneVision and LLaVA-1.5) and evaluating across 18 benchmarks, the approach delivers consistent, significant gains and closes portions of the gap to larger models. Ablation and visualization analyses show that dynamic, context-aware gating and high-density feature injection are crucial for effective fusion, revealing a criss-crossed information flow that enhances grounding, OCR, and complex multimodal reasoning. Overall, CLI establishes a scalable paradigm for restoring symmetry between vision and language in VLMs, enabling deeper multimodal understanding with modest parameter overhead.

Abstract

Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection that links only the output of the vision encoder to the input of the large language model (LLM). This static architecture fundamentally limits the ability of LLMs to achieve comprehensive alignment with hierarchical visual knowledge, compromising their capacity to accurately integrate local details with global semantics into coherent reasoning. To resolve this, we introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities. CLI consists of two synergistic, parameter-efficient components: an Adaptive Multi-Projection (AMP) module that harmonizes features from diverse vision layers, and an Adaptive Gating Fusion (AGF) mechanism that empowers the LLM to selectively inject the most relevant visual information based on its real-time decoding context. We validate the effectiveness and versatility of CLI by integrating it into LLaVA-OneVision and LLaVA-1.5. Extensive experiments on 18 diverse benchmarks demonstrate significant performance improvements, establishing CLI as a scalable paradigm that unlocks deeper multimodal understanding by granting LLMs on-demand access to the full visual hierarchy.

Figures (8)

• Figure 1: (a) An illustrative failure case where our baseline model, relying solely on final-layer features, incorrectly identifies an ice skate as 'roller skates'. This error underscores the necessity of accessing fine-grained details from early vision encoder layers. (b) Visualized gating weights from our CLI framework validate the efficacy of "criss-cross connections". The heatmap reveals that deeper LLM decoder layers (right) dynamically query features from the full spectrum of vision encoder layers (left)---a many-to-many interaction that confirms our proposed fusion architecture.
• Figure 2: (a) The Conventional VLM Pipeline. A vision encoder extracts features from its final layer, which are then mapped into the text embedding space by a projector. (b) Establishes a dynamic "many-to-many" bridge, extracting features from multiple vision layers and adaptively injecting them into multiple layers of the LLM.
• Figure 3: An overview of the proposed Cross-Layer Injection (CLI) Framework. The left panel illustrates the overall "many-to-many"information flow, where hierarchical features from multiple vision encoder layers are dynamically injected into the LLM at multiple decoder layers. The right panel details the core Adaptive Gating Fusion module that enables this process. Multi-Head Attention (MHA) first distills key information from both the incoming visual features and the current LLM hidden state. A gate controller then uses these distilled representations to compute a dynamic weight that governs the selective fusion of the new visual information. This gated process ensures the feature injection is both adaptive and context-sensitive, preventing the LLM from being overwhelmed by irrelevant data.
• Figure 4: Performance gains from CLI on fine-grained visual reasoning. On both OCR and a comprehensive suite of visual grounding benchmarks (RefCOCO/+/g), CLI delivers consistent improvements for both the 0.5B and 7B models. This result demonstrates the framework's effectiveness in leveraging hierarchical visual features for tasks demanding high spatial and textural precision.
• Figure 5: Qualitative comparison of CLI on the LLaVA-OV-7B model across diverse benchmarks. Outputs from Baseline Projector are shown in red, while outputs from CLI are in green. The examples demonstrate that by integrating multi-level visual information, our model achieves more accurate perception and reasoning across various tasks, including MM-Star, LLaVA-in-the-Wild, and OCR-Bench.