Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision

TL;DR

Youtu-VL addresses the loss of fine-grained visual information in Vision-Language Models caused by text-dominant training by introducing Vision-Language Unified Autoregressive Supervision (VLUAS), which treats visual signals as supervisory targets and unifies image and text in a single autoregressive objective. It builds a unified image-text vocabulary via a Synergistic Vision Tokenizer and a vision-language projector, enabling end-to-end training of a standard VLM without task-specific heads and achieving dense visual supervision through an autoregressive framework. The framework demonstrates competitive performance across both general multimodal tasks and vision-centric tasks, supported by extensive multi-stage pre-training on diverse data streams including vision-centric, OCR, STEM, GUI, and synthetic data, along with a robust multi-label NTP-M objective and RL for instruction following. The work provides a practical path toward generalist visual agents by combining perception, reasoning, and alignment in a single architecture, and offers analyses on data scaling, representation, and saturation to guide future scaling. Overall, Youtu-VL advances the goal of unified multimodal intelligence by bridging dense visual perception with language-based reasoning in open-world settings.

Abstract

Despite the significant advancements represented by Vision-Language Models (VLMs), current architectures often exhibit limitations in retaining fine-grained visual information, leading to coarse-grained multimodal comprehension. We attribute this deficiency to a suboptimal training paradigm inherent in prevailing VLMs, which exhibits a text-dominant optimization bias by conceptualizing visual signals merely as passive conditional inputs rather than supervisory targets. To mitigate this, we introduce Youtu-VL, a framework leveraging the Vision-Language Unified Autoregressive Supervision (VLUAS) paradigm, which fundamentally shifts the optimization objective from ``vision-as-input'' to ``vision-as-target.'' By integrating visual tokens directly into the prediction stream, Youtu-VL applies unified autoregressive supervision to both visual details and linguistic content. Furthermore, we extend this paradigm to encompass vision-centric tasks, enabling a standard VLM to perform vision-centric tasks without task-specific additions. Extensive empirical evaluations demonstrate that Youtu-VL achieves competitive performance on both general multimodal tasks and vision-centric tasks, establishing a robust foundation for the development of comprehensive generalist visual agents.

Figures (30)

• Figure 1: Youtu-VL achieves competitive performance on both general multimodal tasks and vision-centric tasks. The concentric rings illustrate the capability scope of different models across various tasks. Colored regions indicate that the model supports the corresponding task, while white regions denote a lack of support. Unlike prior models that exhibit functional gaps, Youtu-VL accommodates a comprehensive range of vision-centric and multimodal tasks via a standard architecture, achieving competitive performance without relying on task-specific modules.
• Figure 2: Comparison between the previous "vision as input" paradigm and the Youtu-VL "vision as target" paradigm. The left panel shows the previous text-dominant VLM, which relies solely on text supervision. The right panel illustrates the Youtu-VL paradigm, which incorporates Vision-Language Unified Autoregressive Supervision (VLUAS), treating vision as a target to achieve unified supervision for both image and text.
• Figure 3: Overview of the Youtu-VL Framework. Left: The architecture integrates a Vision Encoder and Youtu-LLM via a Spatial Merge Projector, operating under the proposed VLUAS paradigm for unified autoregressive modeling. Middle: The Synergistic Vision Tokenizer. We construct a unified vocabulary by fusing semantic and geometric features via cross-attention, optimized with perception and adversarial losses. Right: Dense prediction mechanism. Our proposed NTP-M enables robust multi-label supervision with a relevant negative sampling. Unlike conventional approaches, Youtu-VL achieves direct dense prediction without auxiliary decoders or task-specific tokens.
• Figure 4: The pre-training recipe for Youtu-VL. The top panel illustrates the evolution of the data mixture from Stage 1 to Stage 4: Stages 1 and 2 exclusively utilize pure text data to establish a strong linguistic foundation, while Stages 3 and 4 progressively enhance multimodal capabilities. The bottom panel presents the learning rate schedule aligned with the training stages.
• Figure 5: Data synthesis pipeline for open-world scenarios. The framework processes massive vision-centric data through two parallel branches: (top) object detection and semantic segmentation, utilizing grounding models for raw data and data binding strategies for labeled data; and (bottom) depth estimation, employing depth models and quantization. The pipeline applies specific augmentations to generate a comprehensive dataset for open-world tasks.