OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation

TL;DR

OpenVision 3 tackles the challenge of a single unified visual encoder capable of both understanding and generation by unifying a VAE-based latent space with a trainable ViT, producing a shared representation $z_u$ fed into two branches: reconstruction and understanding. By jointly optimizing $L_{rec}$ and $L_{und}$—where $L_{rec}$ encodes pixel- and latent-level reconstruction and $L_{und}$ combines captioning and contrastive objectives—the model achieves synergistic benefits that improve both generation fidelity and semantic alignment. Across downstream evaluations with the tokenizer frozen, OpenVision 3 rivals CLIP on multimodal understanding benchmarks (e.g., SeedBench, POPE) and surpasses prior unified tokenizers on reconstruction (PSNR, LPIPS, rFID) and generation (gFID) on ImageNet/COCO, illustrating strong cross-task transfer. The work advances unified representation learning and provides a practical path toward unified vision models, with open-source release planned to catalyze future research.

Abstract

This paper presents a family of advanced vision encoder, named OpenVision 3, that learns a single, unified visual representation that can serve both image understanding and image generation. Our core architecture is simple: we feed VAE-compressed image latents to a ViT encoder and train its output to support two complementary roles. First, the encoder output is passed to the ViT-VAE decoder to reconstruct the original image, encouraging the representation to capture generative structure. Second, the same representation is optimized with contrastive learning and image-captioning objectives, strengthening semantic features. By jointly optimizing reconstruction- and semantics-driven signals in a shared latent space, the encoder learns representations that synergize and generalize well across both regimes. We validate this unified design through extensive downstream evaluations with the encoder frozen. For multimodal understanding, we plug the encoder into the LLaVA-1.5 framework: it performs comparably with a standard CLIP vision encoder (e.g., 62.4 vs 62.2 on SeedBench, and 83.7 vs 82.9 on POPE). For generation, we test it under the RAE framework: ours substantially surpasses the standard CLIP-based encoder (e.g., gFID: 1.89 vs 2.54 on ImageNet). We hope this work can spur future research on unified modeling.

Figures (4)

• Figure 1: An overview of OpenVision 3's architecture design and performance highlight. Left panel: The architecture of OpenVision 3. We employ a frozen VAE and a trainable ViT as the unified tokenizer, which produces tokens that are fed simultaneously into both the generation and understanding branches. Middle panel: The learning objectives of the generation branch and the understanding branch. For the generation branch, we focus on high-quality, pixel-level image reconstruction; concurrently, the understanding branch is optimized via joint contrastive learning and captioning objectives. Right panel: The performance summarization shows that OpenVision 3 outperforms other unified tokenizers and semantics-based encoders in rFID and gFID, while remaining competitive with CLIP in multimodal understanding ability.
• Figure 2: Loss visualization with only semantic loss. We trained our tokenizer with and without the reconstruction loss, respectively. In Figures (a) and (b), both pixel-level and latent-level reconstruction losses decrease significantly even in the absence of explicit reconstruction signals. Figures (c) and (d) demonstrate that the incorporation of the reconstruction loss has no adverse impact on the losses of the understanding branch.
• Figure 3: Loss visualization with only reconstruction loss. We trained our tokenizer with and without the understanding loss, respectively. In Figure (a), the inclusion of semantic loss leads to a lower image reconstruction loss, suggesting that semantic supervision can, in turn, enhance reconstruction performance. Figures (c) and (d) reveal that both caption and contrastive losses decrease even without explicit semantic training, further demonstrating that the two objectives are mutually beneficial.
• Figure 4: Qualitative results of class-conditional ImageNet-256 generation. Under the RAE framework, our OpenVision 3 is able to generate high quality images.