DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding

TL;DR

This work tackles the efficiency-cost trade-off in multimodal large language models by revealing redundancy in early layers for cross-modal interactions and proposing DeepInsert, which inserts multimodal tokens into middle layers to bypass early computation. The method is validated across vision, audio, and molecular data with models from hundreds of millions to billions of parameters, achieving 20–25% speedups while preserving or improving accuracy. The authors provide a practical insertion-layer heuristic and RL-based layer selection and demonstrate compatibility with token-pruning approaches, highlighting broad applicability. Overall, the approach offers a scalable, non-intrusive route to deploy efficient multimodal systems and motivates further study of cross-modal processing in deep layers.

Abstract

Hyperscaling of data and parameter count in LLMs is yielding diminishing improvement when weighed against training costs, underlining a growing need for more efficient finetuning and inference without sacrificing performance. This is especially so for multimodal language models (MLMs), where the overhead of processing multimodal tokens can limit their practical viability. Parallely, recent work has uncovered implicit cross-modal alignment in the deeper layers of large MLMs, deepening our understanding of how MLMs process and encode information. Motivated by this, and our observation that MLMs naturally defer most cross-modal token interactions to deeper layers of the model, we propose a simple modification. Instead of concatenation with the language prompt at the start, we insert multimodal tokens directly into the middle, allowing them to entirely bypass the early layers. Our results with diverse modalities, (i) LLaVA \& BLIP for vision, (ii) LTU for audio, and (iii) MoLCA for molecular data, and model sizes, starting from 350M to 13B parameters, indicate that our method reduces both training and inference costs, while at least preserving, if not surpassing the performance of existing baselines.

Figures (12)

• Figure 1: Tradeoff between performance and computational efficiency (inference) in (a) LLaVA v1.5-7B and (b) LTU-7B highlight our two contributions. First, we can reduce multimodal processing (efficiency gains), while either maintaining or improving performance with DeepInsert (see layer 4). Second, it is possible to stay competitive with the baseline, while having significant efficiency gains ($\sim 20$--$25\%$ speedup, see layer 12).
• Figure 2: Attention activity of vision tokens, visualized pre-prediction in Fig. \ref{['fig:attention_all_vision_tokens_separately']} and post-prediction, in Fig. \ref{['fig:var']}, jointly indicate that vision tokens become relevant only after the initial few layers.
• Figure 3: The DeepInsert architecture contrasts with conventional MLLMs: We propose to entirely skip initial layers for the multimodal tokens to exploit underlying redundancies and improve computational efficiency.
• Figure 4: Visualization of intrinsic alignment between pretrained vision (CLIP-based ViT) and language models (Vicuna v1.5), as a function of depth. The vertical axis represents the LLM layer indices, whereas the horizontal axis does the same for the vision encoder. The key point is that for the representations from the latter vision layers (which are generally used as embeddings), alignment becomes stronger as one moves deeper into the LLM.
• Figure 5: Visualization of intrinsic alignment between pretrained vision (DINO-based ViT) and language models (Vicuna v1.5), as a function of depth. The trend is similar to that for CLIP-based models, in for the representations from the latter vision layers (generally used as embeddings), alignment becomes stronger as one moves deeper into the LLM.