Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Depth Completion in Unseen Field Robotics Environments Using Extremely Sparse Depth Measurements

Marco Job, Thomas Stastny, Eleni Kelasidi, Roland Siegwart, Michael Pantic

TL;DR

The paper addresses depth perception for field robotics in unstructured environments by introducing a depth completion approach that combines monocular RGB images with extremely sparse depth measurements, trained entirely on synthetic field-robotics data. Building on a state-of-the-art depth model, it extends the architecture to accept a fourth sparse-depth channel and uses a synthetic dataset pipeline based on SfM-derived textured meshes and photorealistic rendering to learn metric depth in unseen environments. It reports real-time performance on embedded hardware (53 ms per frame on a Nvidia Jetson AGX Orin) and evaluates on five diverse, unseen field-robotics datasets, demonstrating strong generalization and robustness to sparse, noisy measurements. The work also provides a synthetic data generation framework and releases datasets and code to spur further research in field robotics depth perception.

Abstract

Autonomous field robots operating in unstructured environments require robust perception to ensure safe and reliable operations. Recent advances in monocular depth estimation have demonstrated the potential of low-cost cameras as depth sensors; however, their adoption in field robotics remains limited due to the absence of reliable scale cues, ambiguous or low-texture conditions, and the scarcity of large-scale datasets. To address these challenges, we propose a depth completion model that trains on synthetic data and uses extremely sparse measurements from depth sensors to predict dense metric depth in unseen field robotics environments. A synthetic dataset generation pipeline tailored to field robotics enables the creation of multiple realistic datasets for training purposes. This dataset generation approach utilizes textured 3D meshes from Structure from Motion and photorealistic rendering with novel viewpoint synthesis to simulate diverse field robotics scenarios. Our approach achieves an end-to-end latency of 53 ms per frame on a Nvidia Jetson AGX Orin, enabling real-time deployment on embedded platforms. Extensive evaluation demonstrates competitive performance across diverse real-world field robotics scenarios.

Depth Completion in Unseen Field Robotics Environments Using Extremely Sparse Depth Measurements

TL;DR

The paper addresses depth perception for field robotics in unstructured environments by introducing a depth completion approach that combines monocular RGB images with extremely sparse depth measurements, trained entirely on synthetic field-robotics data. Building on a state-of-the-art depth model, it extends the architecture to accept a fourth sparse-depth channel and uses a synthetic dataset pipeline based on SfM-derived textured meshes and photorealistic rendering to learn metric depth in unseen environments. It reports real-time performance on embedded hardware (53 ms per frame on a Nvidia Jetson AGX Orin) and evaluates on five diverse, unseen field-robotics datasets, demonstrating strong generalization and robustness to sparse, noisy measurements. The work also provides a synthetic data generation framework and releases datasets and code to spur further research in field robotics depth perception.

Abstract

Autonomous field robots operating in unstructured environments require robust perception to ensure safe and reliable operations. Recent advances in monocular depth estimation have demonstrated the potential of low-cost cameras as depth sensors; however, their adoption in field robotics remains limited due to the absence of reliable scale cues, ambiguous or low-texture conditions, and the scarcity of large-scale datasets. To address these challenges, we propose a depth completion model that trains on synthetic data and uses extremely sparse measurements from depth sensors to predict dense metric depth in unseen field robotics environments. A synthetic dataset generation pipeline tailored to field robotics enables the creation of multiple realistic datasets for training purposes. This dataset generation approach utilizes textured 3D meshes from Structure from Motion and photorealistic rendering with novel viewpoint synthesis to simulate diverse field robotics scenarios. Our approach achieves an end-to-end latency of 53 ms per frame on a Nvidia Jetson AGX Orin, enabling real-time deployment on embedded platforms. Extensive evaluation demonstrates competitive performance across diverse real-world field robotics scenarios.
Paper Structure (13 sections, 2 equations, 5 figures, 4 tables)

This paper contains 13 sections, 2 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Synthetic Training Dataset Generation
  5. Architecture
  6. Training Loss
  7. Synthetic Depth Measurements at Training Time
  8. Experiments
  9. Evaluation Datasets
  10. Training
  11. Baselines
  12. Results
  13. Conclusions

Figures (5)

  • Figure 1: Our DC approach in five unseen, real-world field robotics environments. The input is a three-channel image and a fourth channel for extremely sparse depth measurements. The blue squares visualize the measurements in this figure. The output is dense metric depth.
  • Figure 2: From an aerial image sequence, using SfM, we obtain a textured 3D mesh of the area. With randomly sampled camera poses, we render synthetic RGB and depth ground truth for training. The input to the model is a four channel-image consisting of RGB and sparse depth (SD) information. The model operates on patches, which are transformed into an embedding vector, and through a ViT and a DPT decoder, we predict metric dense depth maps.
  • Figure 3: From top to bottom row, samples of the Mountain area, Rhône glacier, Road corridor, and Rural area synthetic training datasets. Each pair of columns shows an RGB image and its corresponding metric depth ground truth.
  • Figure 4: At each training step, random corners are sampled from the image (e.g. two corners here), and the corresponding patches are assigned depth values based on the ground truth. The patches are enlarged in this visualization.
  • Figure 5: Latency, defined as the total time required to process input and generate output on an Nvidia 3090 GPU, plotted against the average MAE across all experiments.