Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MT-Depth: Multi-task Instance feature analysis for the Depth Completion

Abdul Haseeb Nizamani, Dandi Zhou, Xinhai Sun

TL;DR

MT-Depth presents an instance-aware depth completion framework that leverages binary instance masks from a frozen YOLO V11 branch to guide depth refinement through cross-attention with a U‑Net backbone. The approach yields superior RMSE and competitive MAE on Virtual KITTI 2, particularly improving depth near object boundaries, occlusions, and thin structures while avoiding dense semantic labels. Key contributions include a novel cross-attention fusion of instance priors with depth features and an attention-guided head that yields refined dense depth maps. The work demonstrates the value of instance-level priors for 3D perception in real-time contexts and motivates further integration of detection and 3D tasks in future depth completion systems.

Abstract

Depth completion plays a vital role in 3D perception systems, especially in scenarios where sparse depth data must be densified for tasks such as autonomous driving, robotics, and augmented reality. While many existing approaches rely on semantic segmentation to guide depth completion, they often overlook the benefits of object-level understanding. In this work, we introduce an instance-aware depth completion framework that explicitly integrates binary instance masks as spatial priors to refine depth predictions. Our model combines four main components: a frozen YOLO V11 instance segmentation branch, a U-Net-based depth completion backbone, a cross-attention fusion module, and an attention-guided prediction head. The instance segmentation branch generates per-image foreground masks that guide the depth branch via cross-attention, allowing the network to focus on object-centric regions during refinement. We validate our method on the Virtual KITTI 2 dataset, showing that it achieves lower Root Mean Squared Error (RMSE) compared to both a U-Net-only baseline and previous semantic-guided methods, while maintaining competitive Mean Absolute Error (MAE). Qualitative and quantitative results demonstrate that the proposed model effectively enhances depth accuracy near object boundaries, occlusions, and thin structures. Our findings suggest that incorporating instance-aware cues offers a promising direction for improving depth completion without relying on dense semantic labels.

MT-Depth: Multi-task Instance feature analysis for the Depth Completion

TL;DR

MT-Depth presents an instance-aware depth completion framework that leverages binary instance masks from a frozen YOLO V11 branch to guide depth refinement through cross-attention with a U‑Net backbone. The approach yields superior RMSE and competitive MAE on Virtual KITTI 2, particularly improving depth near object boundaries, occlusions, and thin structures while avoiding dense semantic labels. Key contributions include a novel cross-attention fusion of instance priors with depth features and an attention-guided head that yields refined dense depth maps. The work demonstrates the value of instance-level priors for 3D perception in real-time contexts and motivates further integration of detection and 3D tasks in future depth completion systems.

Abstract

Depth completion plays a vital role in 3D perception systems, especially in scenarios where sparse depth data must be densified for tasks such as autonomous driving, robotics, and augmented reality. While many existing approaches rely on semantic segmentation to guide depth completion, they often overlook the benefits of object-level understanding. In this work, we introduce an instance-aware depth completion framework that explicitly integrates binary instance masks as spatial priors to refine depth predictions. Our model combines four main components: a frozen YOLO V11 instance segmentation branch, a U-Net-based depth completion backbone, a cross-attention fusion module, and an attention-guided prediction head. The instance segmentation branch generates per-image foreground masks that guide the depth branch via cross-attention, allowing the network to focus on object-centric regions during refinement. We validate our method on the Virtual KITTI 2 dataset, showing that it achieves lower Root Mean Squared Error (RMSE) compared to both a U-Net-only baseline and previous semantic-guided methods, while maintaining competitive Mean Absolute Error (MAE). Qualitative and quantitative results demonstrate that the proposed model effectively enhances depth accuracy near object boundaries, occlusions, and thin structures. Our findings suggest that incorporating instance-aware cues offers a promising direction for improving depth completion without relying on dense semantic labels.

Paper Structure

This paper contains 19 sections, 14 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Instance Segmentation
  4. Depth Completion
  5. Multi‑Task Learning
  6. Method
  7. Segmentation Branch Using YOLO V11
  8. Depth Completion Branch Using U-Net
  9. Cross-Attention
  10. Prediction Head
  11. Overall Model Architecture
  12. Results and Experiment
  13. Virtual KITTI 2 Dataset
  14. Experimental Setup
  15. Evaluation Metrics
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Block Diagram of Proposed MT-Depth Model
  • Figure 2: Qualitative results of our proposed model. (a) RGB input, (b) ground-truth depth, (c) initial prediction, (d) sparse depth, (e) instance mask, and (f) final depth completion.