UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation

TL;DR

The paper identifies conflicting modality-alignment requirements for unified multimodal models and demonstrates that fully shared backbones struggle to satisfy both understanding and generation. It introduces UniFork, a Y-shaped Transformer that shares early layers for cross-task semantic learning and decouples later layers into task-specific branches to balance generation and understanding. Through extensive ablations and benchmarks, UniFork outperforms fully shared architectures and matches or surpasses task-specific experts on both modalities. The work highlights alignment dynamics as a key design principle for scalable, unified multimodal systems and suggests clear paths for scaling and modality expansion.

Abstract

Unified image understanding and generation has emerged as a promising paradigm in multimodal artificial intelligence. Despite recent progress, the optimal architectural design for such unified models remains an open challenge. In this work, we start by analyzing the modality alignment behaviors of task-specific expert models for understanding and generation, as well as current unified models. Our analysis reveals a crucial observation: understanding tasks benefit from a progressively increasing modality alignment across network depth, which helps build up semantic information for better comprehension; In contrast, generation tasks follow a different trend: modality alignment increases in the early layers but decreases in the deep layers to recover spatial details. These divergent alignment patterns create a fundamental conflict in fully shared Transformer backbones, where a uniform representational flow often leads to performance compromises across two tasks. Motivated by this finding, we introduce UniFork, a novel Y-shaped architecture that shares the shallow layers for cross-task representation learning, while employing task-specific branches in deeper layers to avoid task interference. This design effectively balances shared learning and task specialization. Through extensive ablation experiments, we demonstrate that Unifork consistently outperforms conventional fully shared Transformer architectures, and achieves performance on par with or better than task-specific models.

Figures (8)

• Figure 1: Text-to-image generation results by UniFork in 384×384 resolution.
• Figure 2: Modality alignment analysis. We visualize how text-image feature alignment evolves across Transformer layers for both image understanding and generation tasks: (a) Image generation exhibits a rise-then-fall alignment trend across layers. (b) Image understanding shows an increasing alignment pattern. (c) When using a fully shared Transformer for both tasks under the next-token prediction objective, the alignment curves converge, reflecting representational compromise between generation and understanding. (d) Models fine-tuned on Emu3-base wang2024emu3 for each individual task recover their distinct trends, consistent with those observed in expert models.
• Figure 3: Overall framework of UniFork. UniFork adopts a Y-shaped Transformer backbone. The early layers are shared across both image generation and understanding tasks to facilitate joint semantic representation learning, while the later layers are split into task-specific branches to learn specialized representations. Und.: understanding. Gen.: generation. Proj.: projection.
• Figure 4: Three-stage training pipeline for UniFork. The first stage focuses on aligning visual and textual modalities. The second stage performs joint training to enhance both image understanding and generation capabilities. In the third stage, task-specific parameters are alternately optimized using data from each task. Modules involved in training are highlighted in red.
• Figure 5: Modality alignment analysis on MJHQ-30K. The observed alignment patterns on this dataset are consistent with those reported in Section \ref{['section: analysis']}.