Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models

Jingrui Zhang, Feng Liang, Yong Zhang, Wei Wang, Runhao Zeng, Xiping Hu

TL;DR

SparseCut introduces a general cross-modal fusion architecture for multimodal large language models by inserting sparse shortcut connections between a vision encoder and an LLM, enabling multi-level and multi-grained integration of visual features without increasing the LLM's context length. A modality adapter fuse high- and low-resolution visual features before routing through the shortcuts, and a structured shortcut pattern (favoring U-shape, uniform, and sparse connections) optimizes information flow while reducing computational cost. Empirical results across diverse benchmarks show consistent performance gains with robustness to different base LLMs and better handling of visual semantics, particularly in challenging VizWiz-like scenarios where incomplete inputs cause hallucinations. The work demonstrates significant practical benefits for scalable, efficient vision-language understanding and provides reusable code for replication.

Abstract

With the remarkable success of large language models (LLMs) in natural language understanding and generation, multimodal large language models (MLLMs) have rapidly advanced in their ability to process data across multiple modalities. While most existing efforts focus on scaling up language models or constructing higher-quality training data, limited attention has been paid to effectively integrating cross-modal knowledge into the language space. In vision-language models, for instance, aligning modalities using only high-level visual features often discards the rich semantic information present in mid- and low-level features, limiting the model's ability of cross-modality understanding. To address this issue, we propose SparseCut, a general cross-modal fusion architecture for MLLMs, introducing sparse shortcut connections between the cross-modal encoder and the LLM. These shortcut connections enable the efficient and hierarchical integration of visual features at multiple levels, facilitating richer semantic fusion without increasing computational overhead. We further introduce an efficient multi-grained feature fusion module, which performs the fusion of visual features before routing them through the shortcuts. This preserves the original language context and does not increase the overall input length, thereby avoiding an increase in computational complexity for the LLM. Experiments demonstrate that SparseCut significantly enhances the performance of MLLMs across various multimodal benchmarks with generality and scalability for different base LLMs.

Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models

TL;DR

SparseCut introduces a general cross-modal fusion architecture for multimodal large language models by inserting sparse shortcut connections between a vision encoder and an LLM, enabling multi-level and multi-grained integration of visual features without increasing the LLM's context length. A modality adapter fuse high- and low-resolution visual features before routing through the shortcuts, and a structured shortcut pattern (favoring U-shape, uniform, and sparse connections) optimizes information flow while reducing computational cost. Empirical results across diverse benchmarks show consistent performance gains with robustness to different base LLMs and better handling of visual semantics, particularly in challenging VizWiz-like scenarios where incomplete inputs cause hallucinations. The work demonstrates significant practical benefits for scalable, efficient vision-language understanding and provides reusable code for replication.

Abstract

With the remarkable success of large language models (LLMs) in natural language understanding and generation, multimodal large language models (MLLMs) have rapidly advanced in their ability to process data across multiple modalities. While most existing efforts focus on scaling up language models or constructing higher-quality training data, limited attention has been paid to effectively integrating cross-modal knowledge into the language space. In vision-language models, for instance, aligning modalities using only high-level visual features often discards the rich semantic information present in mid- and low-level features, limiting the model's ability of cross-modality understanding. To address this issue, we propose SparseCut, a general cross-modal fusion architecture for MLLMs, introducing sparse shortcut connections between the cross-modal encoder and the LLM. These shortcut connections enable the efficient and hierarchical integration of visual features at multiple levels, facilitating richer semantic fusion without increasing computational overhead. We further introduce an efficient multi-grained feature fusion module, which performs the fusion of visual features before routing them through the shortcuts. This preserves the original language context and does not increase the overall input length, thereby avoiding an increase in computational complexity for the LLM. Experiments demonstrate that SparseCut significantly enhances the performance of MLLMs across various multimodal benchmarks with generality and scalability for different base LLMs.
Paper Structure (20 sections, 4 equations, 4 figures, 6 tables)

This paper contains 20 sections, 4 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models
  4. Multimodal Large Language Models
  5. Method
  6. Overview
  7. Shortcut Connections
  8. Efficient Multi-Grained Fusion
  9. Experiment
  10. Implementation and Experiment Setup
  11. Main Results
  12. Ablation Study
  13. Effects of various shortcut connection patterns.
  14. Effects of multi-grained fusion of visual features.
  15. Alternative Base LLMs
  16. ...and 5 more sections

Figures (4)

  • Figure 1: (a) The overview of the SparseCut mechanism. It supports multi-level cross-modal fusion across different model layers, as well as high- and low-resolution image fusion through shortcut connections. (b) Performance of SparseCut on various multimodal benchmarks.
  • Figure 2: Various shortcut connection patterns
  • Figure 3: A comparison of different multi-grained visual fusion in an MLLM. (a) Traditional fusion: Conventional MLLMs handle high-resolution features by simply concatenating them with low-resolution ones along the token dimension, resulting in an image token sequence that is five times longer. (b) Shortcut fusion: Multi-resolution features fuse in each shortcut to the LLM, reducing the visual context length by four times. The blank token corresponds to textual tokens, meaning that no operation is performed on original textual tokens at this stage.
  • Figure 4: Case study examples on the VizWiz benchmark.