Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models

TL;DR

The paper identifies gradient conflicts in fully modality-shared autoregressive unified multimodal models caused by distinct low-level statistics of vision and text. It proposes Uni-X, a two-end-separated, middle-shared transformer that allocates modality-specific processing at the ends and a shared semantic core in the middle. Through extensive scaling and ablation studies, Uni-X demonstrates improved training efficiency and competitive performance, with a 3B Uni-X matching or surpassing some 7B AR-based UMMs in text, vision understanding, and image generation (GenEval score of 82). The work offers a scalable, parameter-efficient foundation for future unified multimodal modeling and provides the codebase for reproducibility.

Abstract

Unified Multimodal Models (UMMs) built on shared autoregressive (AR) transformers are attractive for their architectural simplicity. However, we identify a critical limitation: when trained on multimodal inputs, modality-shared transformers suffer from severe gradient conflicts between vision and text, particularly in shallow and deep layers. We trace this issue to the fundamentally different low-level statistical properties of images and text, while noting that conflicts diminish in middle layers where representations become more abstract and semantically aligned. To overcome this challenge, we propose Uni-X, a two-end-separated, middle-shared architecture. Uni-X dedicates its initial and final layers to modality-specific processing, while maintaining shared parameters in the middle layers for high-level semantic fusion. This X-shaped design not only eliminates gradient conflicts at both ends but also further alleviates residual conflicts in the shared layers. Extensive experiments validate the effectiveness of Uni-X. Under identical training conditions, Uni-X achieves superior training efficiency compared to strong baselines. When scaled to 3B parameters with larger training data, Uni-X matches or surpasses 7B AR-based UMMs, achieving a GenEval score of 82 for image generation alongside strong performance in text and vision understanding tasks. These results establish Uni-X as a parameter-efficient and scalable foundation for future unified multimodal modeling. Our code is available at https://github.com/CURRENTF/Uni-X

Figures (10)

• Figure 1: Gradient conflict analysis of down-projection weights in the FFN of a modality-shared transformer. The shared transformer exhibits severe conflicts in shallow and deep layers, with only partial mitigation in intermediate layers. In contrast, Uni-X avoids conflicts at both extremes and further alleviates them in the middle layers.
• Figure 2: Illustration of gradient conflict. (a) The loss landscapes of different modalities exhibit distinct geometries, creating potential conflicts in optimization direction. (b) The optimum of the sum of losses is different from the optimum of any single modality's loss. (c) In the presence of gradient conflict, the optimization trajectory becomes oscillating and suffers from slow convergence.
• Figure 3: Conditional entropy of images and natural languages. Image token sequences encoded by the VQ tokenizer exhibit substantially higher entropy, indicating greater difficulty in prediction.
• Figure 4: Illustration of the proposed Uni-X architecture compared with a standard modality-shared transformer. The baseline shared transformer (left) encounters gradient conflicts in shallow and deep layers due to the mismatched statistical properties of vision and text tokens. In contrast, Uni-X (right) adopts a two-end-separated, middle-shared design: modality-specific layers at both ends handle low-level feature processing, while a shared central block performs high-level semantic fusion. This structure aligns the architecture with the inherent characteristics of each modality and effectively mitigates gradient conflicts.
• Figure 5: Qualitative examples of Uni-X image generation. The results highlight its ability to produce diverse, high-quality visuals that follow prompts with both creativity and fine-grained detail.