HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenization

Abstract

Unified Multimodal Models struggle to bridge the fundamental gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. Existing approaches typically compromise by employing decoupled encoders, stacking representation encoder atop VAEs, or utilizing discrete quantization. However, these methods often disrupt information coherence and lead to optimization conflicts. To this end, we introduce HYDRA-TOK, a representation-harmonized pure ViT in the insight that visual modeling should evolve from generation to understanding. HYDRA-TOK reformulates the standard backbone into a progressive learner that transitions from a Gen-ViT, which captures structure-preserving primitives, to a Sem-ViT for semantic encoding. Crucially, this transition is mediated by a Generation-Semantic Bottleneck (GSB), which compresses features into a low-dimensional space to filter noise for robust synthesis, then restores dimensionality to empower complex semantic comprehension. Built upon this foundation, we present HYDRA, a native unified framework integrating perception and generation within a single parameter space. Extensive experiments establish HYDRA as a new state-of-the-art. It sets a benchmark in visual reconstruction (rFID 0.08) and achieves top-tier generation performance on GenEval (0.86), DPG-Bench (86.4), and WISE (0.53), while simultaneously outperforming previous native UMMs by an average of 10.0 points across eight challenging understanding benchmarks.

Figures (19)

• Figure 0: Multimodal understanding and image generation results from HYDRA. (a) Our model outperforms previous state-of-the-art unified multimodal models as well as several task-specific models across diverse benchmarks. (b) Our model demonstrates robust visual generation capabilities, producing high-fidelity images with accurate semantic alignment.
• Figure 1: Representation schemes in native unified multimodal models. (a) Decoupled Encoder bageljanus: It employs a VAE and a representation encoder as dedicated encoders for generation and understanding tasks, respectively. (b) Sequential Encoder xie2025show: It feeds the output of the VAE directly into the representation encoder in a cascaded manner. (c) Single-representation encoder ma2025unitokwu2025harmonizing: It adopts a standalone representation encoder to unify representation learning for both understanding and generation tasks. (d) Our Proposed Representation-Harmonized ViT Design: it also leverages a single ViT backbone, while introducing a bottleneck module to harmonize the feature learning processes of understanding and generation tasks.
• Figure 2: Training process illustration for HYDRA-TOK and HYDRA. (a) HYDRA-TOK functions as a progressive learner, bridging the gap between reconstruction and understanding. It employs a Generation-Semantic Bottleneck (GSB) to execute a unique compress-and-reconstruct operation, effectively filtering noise to transition from structure-preserving primitives (Gen-ViT) to semantic abstractions (Sem-ViT). (b) HYDRA achieves representational unification upon this foundation, utilizing a dual-head mechanism to seamlessly integrate autoregressive text prediction with rectified flow matching for image generation.
• Figure 3: Ablation results on different latent channel dimensions.
• Figure 4: Ablation results on different layer configurations.