X-Fusion: Introducing New Modality to Frozen Large Language Models

TL;DR

X-Fusion presents a practical framework for extending frozen pretrained LLMs with a separate trainable vision tower to achieve unified vision-language capabilities without degrading language performance. By freezing the language backbone and introducing modality-specific layers, the model demonstrates strong performance on both image understanding (I2T) and image generation (T2I) tasks, supported by data-centric ablations that reveal how noise, data ratios, and feature alignment affect outcomes. Key contributions include the dual-tower architecture, comprehensive architectural ablations, and extensions such as the X-Fuse layer and transfer from pretrained diffusion models, which collectively enable efficient, scalable multimodal learning. The work offers actionable guidance for building practical unified multimodal models with implications for accessibility, content creation, and vision-language reasoning.

Abstract

We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks while preserving their language capabilities. X-Fusion employs a dual-tower design with modality-specific weights, keeping the LLM's parameters frozen while integrating vision-specific information for both understanding and generation. Our experiments demonstrate that X-Fusion consistently outperforms alternative architectures on both image-to-text and text-to-image tasks. We find that incorporating understanding-focused data improves generation quality, reducing image data noise enhances overall performance, and feature alignment accelerates convergence for smaller models but has minimal impact on larger ones. Our findings provide valuable insights into building efficient unified multimodal models.

Figures (16)

• Figure 1: We introduce X-Fusion - a novel framework that adapts pretrained LLMs (e.g., LLaMA) to new modalities (e.g., vision) while retaining their language capabilities and world knowledge.
• Figure 2: Captions generated by X-Fusion demonstrate high details and strong visual alignment with the image inputs.
• Figure 3: Images generated by X-Fusion demonstrate high visual quality and strong text alignment with the input prompts.
• Figure 4: Conceptual comparison of four model architecture baselines. Here, we illustrate how each layer processes the sequential multi-modal feature. (a) Single Tower: Directly fine-tuning pre-trained LLM. (b) Gated Layer: Duplicate Each LLM layer as the gated vision layer, (c) Dual Projection: Duplicate QKV matric and MLP layer for vision modality, (d) Dual Tower: Duplicated transformer block for vision modality.
• Figure 5: Performance of image generation and understanding at various data ratios. Increasing visual understanding data improves visual generation performance.