Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss

Bozhou Li, Xinda Xue, Sihan Yang, Yang Shi, Xinlong Chen, Yushuo Guan, Yuanxing Zhang, Wentao Zhang

TL;DR

This work reveals a hidden instability in Pre-Norm Multimodal LLMs: a persistent norm gap between high-norm visual tokens and low-norm text tokens induces an asymmetric update dynamic that hampers cross-modal fusion. The authors develop a theoretical framework predicting slower evolution of visual representations and faster textual adaptation, leading to degraded attention and information transfer across modalities. They validate this dynamic across multiple open-source MLLMs and propose a simple yet effective solution: insert a LayerNorm after the visual projector with Global Weight Compensation to preserve gradient flow. The results show significant gains on multimodal benchmarks and notable improvements on text-only tasks, suggesting that aligning norms at the modality interface yields a more holistically capable model. This work provides a practical design principle for robust cross-modal fusion in large multimodal models.

Abstract

Multimodal Large Language Models (MLLMs), which couple pre-trained vision encoders and language models, have shown remarkable capabilities. However, their reliance on the ubiquitous Pre-Norm architecture introduces a subtle yet critical flaw: a severe norm disparity between the high-norm visual tokens and the low-norm text tokens. In this work, we present a formal theoretical analysis demonstrating that this imbalance is not a static issue. Instead, it induces an ``asymmetric update dynamic,'' where high-norm visual tokens exhibit a ``representational inertia,'' causing them to transform semantically much slower than their textual counterparts. This fundamentally impairs effective cross-modal feature fusion. Our empirical validation across a range of mainstream MLLMs confirms that this theoretical dynamic -- the persistence of norm disparity and the resulting asymmetric update rates -- is a prevalent phenomenon. Based on this insight, we propose a remarkably simple yet effective solution: inserting a single, carefully initialized LayerNorm layer after the visual projector to enforce norm alignment. Experiments conducted on the LLaVA-1.5 architecture show that this intervention yields significant performance gains not only on a wide suite of multimodal benchmarks but also, notably, on text-only evaluations such as MMLU, suggesting that resolving the architectural imbalance leads to a more holistically capable model.

The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss

TL;DR

This work reveals a hidden instability in Pre-Norm Multimodal LLMs: a persistent norm gap between high-norm visual tokens and low-norm text tokens induces an asymmetric update dynamic that hampers cross-modal fusion. The authors develop a theoretical framework predicting slower evolution of visual representations and faster textual adaptation, leading to degraded attention and information transfer across modalities. They validate this dynamic across multiple open-source MLLMs and propose a simple yet effective solution: insert a LayerNorm after the visual projector with Global Weight Compensation to preserve gradient flow. The results show significant gains on multimodal benchmarks and notable improvements on text-only tasks, suggesting that aligning norms at the modality interface yields a more holistically capable model. This work provides a practical design principle for robust cross-modal fusion in large multimodal models.

Abstract

Multimodal Large Language Models (MLLMs), which couple pre-trained vision encoders and language models, have shown remarkable capabilities. However, their reliance on the ubiquitous Pre-Norm architecture introduces a subtle yet critical flaw: a severe norm disparity between the high-norm visual tokens and the low-norm text tokens. In this work, we present a formal theoretical analysis demonstrating that this imbalance is not a static issue. Instead, it induces an ``asymmetric update dynamic,'' where high-norm visual tokens exhibit a ``representational inertia,'' causing them to transform semantically much slower than their textual counterparts. This fundamentally impairs effective cross-modal feature fusion. Our empirical validation across a range of mainstream MLLMs confirms that this theoretical dynamic -- the persistence of norm disparity and the resulting asymmetric update rates -- is a prevalent phenomenon. Based on this insight, we propose a remarkably simple yet effective solution: inserting a single, carefully initialized LayerNorm layer after the visual projector to enforce norm alignment. Experiments conducted on the LLaVA-1.5 architecture show that this intervention yields significant performance gains not only on a wide suite of multimodal benchmarks but also, notably, on text-only evaluations such as MMLU, suggesting that resolving the architectural imbalance leads to a more holistically capable model.

Paper Structure

This paper contains 48 sections, 1 theorem, 22 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Self-Attention
  4. Normalization Layers in Transformers
  5. Layer Normalization (LayerNorm).
  6. Root Mean Square Norm (RMSNorm).
  7. The Pre-Norm vs. Post-Norm Residual Architecture
  8. Post-Norm Architecture.
  9. Pre-Norm Architecture.
  10. Theoretical Analysis of Norm-Induced Decoupling Effect
  11. Analytical Framework and Assumptions
  12. Asymmetric Angular Velocity and Geometric Divergence
  13. Signal Degradation and Attention Collapse
  14. Empirical Validation: Probing the Dynamics of Norm Imbalance
  15. Experiments
  16. ...and 33 more sections

Key Result

Lemma 1

For a fixed total update potential (represented by the geometric mean of angular velocities), the similarity retention factor $\gamma = \cos(\theta_1)\cos(\theta_2)$ is maximized when velocities are symmetric ($\theta_1 = \theta_2$). Conversely, asymmetry strictly decreases $\gamma$.

Figures (5)

  • Figure 3: A comparison of token dynamics with and without our norm alignment method. (a) shows the layer-wise L2 norm evolution, while (b) shows the inter-layer cosine similarity, which acts as a proxy for update rate.
  • Figure 4: Evolution of layer-wise cosine similarity across different checkpoints during the multimodal pre-training phase (w Norm).
  • Figure :
  • Figure :
  • Figure :

Theorems & Definitions (2)

  • Lemma 1: Asymmetry Maximizes Decay Rate
  • proof