UniViTAR: Unified Vision Transformer with Native Resolution

TL;DR

UniViTAR introduces a homogeneous vision backbone designed to process images and videos at native resolutions with dynamic aspect ratios. It combines architectural upgrades (2D RoPE, SwiGLU, RMSNorm, QK-Norm, LayerScale) with a four-stage progressive training regime that includes resolution curriculum learning, inter-batch image-video switching, and sigmoid-based contrastive learning plus distillation. Across 0.3B to 1B parameter scales, UniViTAR delivers strong zero-shot image/video classification and retrieval, competitive linear probing and dense prediction results, and superior multimodal understanding when used as a vision encoder for MLLMs. The framework relies solely on public data and demonstrates that native-resolution modeling with modular training strategies yields robust, scalable visual representations for multimodal tasks with practical efficiency.

Abstract

Conventional Vision Transformer simplifies visual modeling by standardizing input resolutions, often disregarding the variability of natural visual data and compromising spatial-contextual fidelity. While preliminary explorations have superficially investigated native resolution modeling, existing approaches still lack systematic analysis from a visual representation perspective. To bridge this gap, we introduce UniViTAR, a family of homogeneous vision foundation models tailored for unified visual modality and native resolution scenario in the era of multimodal. Our framework first conducts architectural upgrades to the vanilla paradigm by integrating multiple advanced components. Building upon these improvements, a progressive training paradigm is introduced, which strategically combines two core mechanisms: (1) resolution curriculum learning, transitioning from fixed-resolution pretraining to native resolution tuning, thereby leveraging ViT's inherent adaptability to variable-length sequences, and (2) visual modality adaptation via inter-batch image-video switching, which balances computational efficiency with enhanced temporal reasoning. In parallel, a hybrid training framework further synergizes sigmoid-based contrastive loss with feature distillation from a frozen teacher model, thereby accelerating early-stage convergence. Finally, trained exclusively on public datasets, externsive experiments across multiple model scales from 0.3B to 1B demonstrate its effectiveness.

Figures (6)

• Figure 1: An overview of the UniViTAR’s scaling behavior on a broad range of tasks (left) and its superior downstream multimodal performance (right) across diverse benchmarks.
• Figure 2: Brief illustration of UniViTAR family pipeline. UniViTAR supports processing input at its native resolution, and also supports scaling the resolution down or up while maintaining the aspect ratio to accommodate different application scenarios, such as higher computational efficiency or finer-grained visual details. By treating video inputs as temporally extended image sequences, the framework uniformly produces longer variable-length visual token sequences.
• Figure 3: Architecture of UniViTAR family. All visual inputs are uniformly transformed into patch sequences and fed into Vision Transformer. In addition to using the Pre-Norm approach, we also adopt RMS-Norm as the normalization layer in both MHA and FFN module. Finally, a variable-length visual token sequence is produced. Best viewed on screen.
• Figure 4: Performance comparison of different resolution modes as the length of the vision sequence increases. The black dashed line shows the performance when using native resolution.
• Figure 5: Average performance improvement illustration across different training stages.