Rethinking Visual Information Processing in Multimodal LLMs

TL;DR

Rethinking Visual Information Processing in Multimodal LLMs addresses the modality gap between vision and language in MLLMs by letting the LLM itself act as a vision encoder. The authors propose LLaViT, which adds separate QKV projections for visual tokens, bidirectional attention among visual tokens, and a fusion of local and global CLIP features, enabling richer visual representations inside the LLM. Across 17 benchmarks, LLaViT substantially surpasses the LLaVA baseline and even matches or beats models with much larger parameter counts, demonstrating a new paradigm for MLLM design. This approach reduces the parameter gap while achieving strong vision-centric and OCR capabilities, with notable gains in fine-grained visual understanding.

Abstract

Despite the remarkable success of the LLaVA architecture for vision-language tasks, its design inherently struggles to effectively integrate visual features due to the inherent mismatch between text and vision modalities. We tackle this issue from a novel perspective in which the LLM not only serves as a language model but also a powerful vision encoder. To this end, we present LLaViT - Large Language Models as extended Vision Transformers - which enables the LLM to simultaneously function as a vision encoder through three key modifications: (1) learning separate QKV projections for vision modality, (2) enabling bidirectional attention on visual tokens, and (3) incorporating both global and local visual representations. Through extensive controlled experiments on a wide range of LLMs, we demonstrate that LLaViT significantly outperforms the baseline LLaVA method on a multitude of benchmarks, even surpassing models with double its parameter count, establishing a more effective approach to vision-language modeling.

Figures (6)

• Figure 1: Visualizing how the LLM interprets visual tokens at the input and output layers of the LLM. Input layer word representations are selected using Eq. \ref{['eq:cosine_sim']}. Output layer word representations are selected based on the LLM's final logits of the visual token. For better interpretability, we manually select 2 of the top-$3$ word representations for each of the selected visual tokens.
• Figure 2: Overview of LLaViT, which transforms the LLM to act as an extended vision encoder. (a) We learn separate QKV projection parameters for visual tokens, initialized with the weights of the LLM's QKV parameters. (b) While the LLM employs causal attention on all tokens, we enable bidirectional attention on the visual tokens. (c) We incorporate both local and global features in the visual tokens by extracting patch features from multiple layers of the CLIP ViT model.
• Figure 3: Output logit visualizations for (a) Qwen2.5-3B and (b) -7B models, comparing LLaViT with the baseline. For each visual token, we extract the top-$3$ words and display the two most sensible words, filtering irrelevant symbols such as '\\ n' or punctuations. Words are shown in bold if they are relevant to the corresponding image patch.
• Figure 4: Detailed version of Figure 1, showing the raw Top-$3$ input and output words, alongside their cosine similarity or output probabilities. Non-alphabetic characters are displayed in their unicode representations, and we highlight semantically relevant words in yellow.
• Figure 5: Visualizing the output of visual tokens, comparing the baseline and LLaViT using Qwen2.5-3B as the base LLM. We visualize the top-$3$ outputs for each model alongside the output probabilities, and highlight semantically relevant words in yellow.