Revisiting Multi-Task Visual Representation Learning

TL;DR

This paper addresses the gap between global semantic alignment and fine-grained spatial reasoning in visual representation learning by proposing MTV, a unified multi-task pretraining framework. MTV jointly optimizes a shared Vision Transformer across vision-language contrastive, self-supervised, and dense spatial objectives, using expert-model pseudo-labels for grounding and depth supervision. Across 100M-pair pretraining and diverse downstream tasks, MTV demonstrates consistent gains in both semantic and geometric understanding, achieving 69.4% zero-shot ImageNet accuracy for ViT-Base and superior performance on depth, segmentation, and correspondence tasks compared with VL-only baselines. The work highlights favorable task synergy, data efficiency, and scalable pathways toward general-purpose visual encoders, with potential extensions to larger data regimes and multimodal modalities.

Abstract

Current visual representation learning remains bifurcated: vision-language models (e.g., CLIP) excel at global semantic alignment but lack spatial precision, while self-supervised methods (e.g., MAE, DINO) capture intricate local structures yet struggle with high-level semantic context. We argue that these paradigms are fundamentally complementary and can be integrated into a principled multi-task framework, further enhanced by dense spatial supervision. We introduce MTV, a multi-task visual pretraining framework that jointly optimizes a shared backbone across vision-language contrastive, self-supervised, and dense spatial objectives. To mitigate the need for manual annotations, we leverage high-capacity "expert" models -- such as Depth Anything V2 and OWLv2 -- to synthesize dense, structured pseudo-labels at scale. Beyond the framework, we provide a systematic investigation into the mechanics of multi-task visual learning, analyzing: (i) the marginal gain of each objective, (ii) task synergies versus interference, and (iii) scaling behavior across varying data and model scales. Our results demonstrate that MTV achieves "best-of-both-worlds" performance, significantly enhancing fine-grained spatial reasoning without compromising global semantic understanding. Our findings suggest that multi-task learning, fueled by high-quality pseudo-supervision, is a scalable path toward more general visual encoders.

Figures (4)

• Figure 1: Multi-task supervision effects on a ViT-L model with 10M training samples. We integrate VL, SSL, and pseudo-labeled grounding and depth estimation into a unified training framework. This leads to strong and consistent performance gains across diverse vision and vision–language tasks.
• Figure 2: Overview of our MTV framework.(a) Each image is paired with a web-crawled caption, and augmented with pseudo region–text pairs and relative depth maps generated by teacher models. (b) MTV jointly learns from three complementary supervision types: Global (image–caption contrast), Dense (region–text alignment, depth), and SSL (self-distillation, masked feature prediction). A shared image encoder is optimized together with a text encoder and an EMA teacher.
• Figure 3: Data scaling behavior of multi-task visual pretraining.ViT-Base models are trained with 10M, 50M, and 100M samples under our multi-task setting. CLIP-Baseclip is shown as gray dashed line. Lower $\downarrow$ is better for NYUv2; higher $\uparrow$ is better elsewhere.
• Figure 4: Visualizations of zero-shot relative depth estimation.