EDADepth: Enhanced Data Augmentation for Monocular Depth Estimation

TL;DR

A novel EDADepth, an enhanced data augmentation method to estimate monocular depth without using additional training data, and achieves state-of-the-art results (SOTA) on the $\delta_{3}$ metric on the NYUv2 and KITTI datasets.

Abstract

Due to their text-to-image synthesis feature, diffusion models have recently seen a rise in visual perception tasks, such as depth estimation. The lack of good-quality datasets makes the extraction of a fine-grain semantic context challenging for the diffusion models. The semantic context with fewer details further worsens the process of creating effective text embeddings that will be used as input for diffusion models. In this paper, we propose a novel EDADepth, an enhanced data augmentation method to estimate monocular depth without using additional training data. We use Swin2SR, a super-resolution model, to enhance the quality of input images. We employ the BEiT pre-trained semantic segmentation model for better extraction of text embeddings. We use BLIP-2 tokenizer to generate tokens from these text embeddings. The novelty of our approach is the introduction of Swin2SR, the BEiT model, and the BLIP-2 tokenizer in the diffusion-based pipeline for the monocular depth estimation. Our model achieves state-of-the-art results (SOTA) on the delta3 metric on NYUv2 and KITTI datasets. It also achieves results comparable to those of the SOTA models in the RMSE and REL metrics. Finally, we also show improvements in the visualization of the estimated depth compared to the SOTA diffusion-based monocular depth estimation models. Code: https://github.com/edadepthmde/EDADepth_ICMLA.

Figures (7)

• Figure 1: Our EDADepth model takes single image (top) and estimates depth using a pre-trained U-Net model. It uses the BEiT semantic segmentation model to extract context for the generation of depth maps (Middle). 3D point cloud (Bottom) is constructed from the estimated depth map and the respective input RGB image.
• Figure 2: In EDADepth, the raw RGB input image is enhanced using a Swin2SR model. BEiT model extracts detailed semantic context from the enhanced image and passes it to a BLIP-2 tokenizer for tokens. These text embedding tokens are fed to a pre-trained U-Net model to estimate depth.
• Figure 3: EDADepth model framework. The architecture integrates a Swin2SR model to process raw RGB inputs, producing enhanced images for the text embedding module. It utilizes the BEiT semantic segmentation model for a segmentation-based, self-supervised text embedding process that generates a vector of text embeddings. These vectors are then fed into the U-Net model via the BLIP-2 tokenizer. The model follows a forward-reverse denoising process to generate an estimated depth map.
• Figure 4: Text-embedding extraction using BEiT model.
• Figure 5: Probabilities of the predicted semantic classes for the original, the resized, and the super-resolved images.