On the Viability of Monocular Depth Pre-training for Semantic Segmentation

TL;DR

This paper investigates whether pre-training a model on monocular depth can improve downstream semantic segmentation, motivated by reducing human annotation bias and enabling domain-relevant, geometry-based learning. It formalizes the question via an information-theoretic framework and conducts an extensive set of experiments across supervision forms (structure-from-motion, stereo, depth sensors, and monocular video), architectures, and datasets (KITTI, Cityscapes, NYU-V2), comparing against ImageNet and random initializations. The key finding is that depth pre-training consistently enhances semantic segmentation performance and speeds up training, with depth supervision (LiDAR and stereo) often providing the strongest gains and optical flow lagging behind. Out-of-domain transfer experiments with DepthAnything demonstrate robust cross-domain improvements on ADE20k, Pascal VOC, and Cityscapes, supporting the practical viability of depth-based pre-training as a cheaper, less biased alternative to large-scale supervised pre-training, while highlighting calibration and domain considerations for real-world deployment.

Abstract

The question of whether pre-training on geometric tasks is viable for downstream transfer to semantic tasks is important for two reasons, one practical and the other scientific. If the answer is positive, we may be able to reduce pre-training cost and bias from human annotators significantly. If the answer is negative, it may shed light on the role of embodiment in the emergence of language and other cognitive functions in evolutionary history. To frame the question in a way that is testable with current means, we pre-train a model on a geometric task, and test whether that can be used to prime a notion of 'object' that enables inference of semantics as soon as symbols (labels) are assigned. We choose monocular depth prediction as the geometric task, and semantic segmentation as the downstream semantic task, and design a collection of empirical tests by exploring different forms of supervision, training pipelines, and data sources for both depth pre-training and semantic fine-tuning. We find that monocular depth is a viable form of pre-training for semantic segmentation, validated by improvements over common baselines. Based on the findings, we propose several possible mechanisms behind the improvements, including their relation to dataset size, resolution, architecture, in/out-of-domain source data, and validate them through a wide range of ablation studies. We also find that optical flow, which at first glance may seem as good as depth prediction since it optimizes the same photometric reprojection error, is considerably less effective, as it does not explicitly aim to infer the latent structure of the scene, but rather the raw phenomenology of temporally adjacent images.

Figures (15)

• Figure 1: Diagram for different pre-training and fine-tuning setups. (a) Common practice: pre-train the encoder, e.g. on ImageNet, attach a decoder, and fine-tune the network. (b) Our best practice: pre-train the network by monocular depth, and fine-tune for semantic segmentation. (c) Cross architecture: for fair comparisons with common practice, we pre-train by depth, replace the decoder, and fine-tune. (d) To test the quality of pre-trained encoders, we fix the encoders and fine-tune decoders only.
• Figure 2: Comparison between different network initializations. Models initialized by depth pre-training (unsupervised) train faster and achieve higher final accuracy.
• Figure 3: Final accuracy vs different training set size. Under all training set sizes, our best practice constantly outperforms ImageNet pre-trained. Encoder: ResNet 18.
• Figure 3: Initializing with depth encoder and random decoder. Initializing with the depth encoder and a random decoder outperforms ImageNet initialization, but is worse than initializing with both encoder and decoder from the depth network (see Tab. \ref{['tab:results']}).
• Figure 4: Frozen encoder results. Using an encoder pre-trained by depth significantly outperforms one with random weights and one for ImageNet classification. Note that in this experiment, ImageNet pre-training performs worse than random initialization.