Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAID-NeRF: Segmentation-AIDed NeRF for Depth Completion of Transparent Objects

Avinash Ummadisingu, Jongkeum Choi, Koki Yamane, Shimpei Masuda, Naoki Fukaya, Kuniyuki Takahashi

TL;DR

This paper proposes using Visual Foundation Models for segmentation in a zero-shot, label-free way to guide the NeRF reconstruction process for these objects via the simultaneous reconstruction of semantic fields and extensions to increase robustness.

Abstract

Acquiring accurate depth information of transparent objects using off-the-shelf RGB-D cameras is a well-known challenge in Computer Vision and Robotics. Depth estimation/completion methods are typically employed and trained on datasets with quality depth labels acquired from either simulation, additional sensors or specialized data collection setups and known 3d models. However, acquiring reliable depth information for datasets at scale is not straightforward, limiting training scalability and generalization. Neural Radiance Fields (NeRFs) are learning-free approaches and have demonstrated wide success in novel view synthesis and shape recovery. However, heuristics and controlled environments (lights, backgrounds, etc) are often required to accurately capture specular surfaces. In this paper, we propose using Visual Foundation Models (VFMs) for segmentation in a zero-shot, label-free way to guide the NeRF reconstruction process for these objects via the simultaneous reconstruction of semantic fields and extensions to increase robustness. Our proposed method Segmentation-AIDed NeRF (SAID-NeRF) shows significant performance on depth completion datasets for transparent objects and robotic grasping.

SAID-NeRF: Segmentation-AIDed NeRF for Depth Completion of Transparent Objects

TL;DR

This paper proposes using Visual Foundation Models for segmentation in a zero-shot, label-free way to guide the NeRF reconstruction process for these objects via the simultaneous reconstruction of semantic fields and extensions to increase robustness.

Abstract

Acquiring accurate depth information of transparent objects using off-the-shelf RGB-D cameras is a well-known challenge in Computer Vision and Robotics. Depth estimation/completion methods are typically employed and trained on datasets with quality depth labels acquired from either simulation, additional sensors or specialized data collection setups and known 3d models. However, acquiring reliable depth information for datasets at scale is not straightforward, limiting training scalability and generalization. Neural Radiance Fields (NeRFs) are learning-free approaches and have demonstrated wide success in novel view synthesis and shape recovery. However, heuristics and controlled environments (lights, backgrounds, etc) are often required to accurately capture specular surfaces. In this paper, we propose using Visual Foundation Models (VFMs) for segmentation in a zero-shot, label-free way to guide the NeRF reconstruction process for these objects via the simultaneous reconstruction of semantic fields and extensions to increase robustness. Our proposed method Segmentation-AIDed NeRF (SAID-NeRF) shows significant performance on depth completion datasets for transparent objects and robotic grasping.
Paper Structure (25 sections, 4 equations, 6 figures, 5 tables, 1 algorithm)

This paper contains 25 sections, 4 equations, 6 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transparent Object Segmentation
  4. Learning-based Depth Completion
  5. Neural Radiance Fields (NeRFs)
  6. Robotic Grasping with NeRF
  7. Method
  8. Preliminary: Neural Radiance Fields (NeRFs)
  9. Semantic Fields as NeRF Density Grounding
  10. Position Encoding
  11. Detection & Semantic Mask Generation
  12. Robotic Grasp Planning
  13. Experiment: Depth Completion Dataset
  14. Experiment Setup
  15. Implementation Details
  16. ...and 10 more sections

Figures (6)

  • Figure 1: Overview of the proposed system for robotic grasping using Segmentation-AIDed NeRF (SAID-NeRF) for depth estimation of transparent objects. SAID-NeRF uses Visual Foundation Models (VFMs) for zero-shot segmentation to guide the reconstruction process
  • Figure 2: Architecture diagram of the proposed SAID-NeRF
  • Figure 3: Samples from the ClearPose dataset chen2022clearpose in a cluttered setting, with opaque distractors, a translucent cover, and non-planar settings
  • Figure 4: Visualization of completed depth on scenes from "New Background" and "Heavy Occlusion" from the ClearPose chen2022clearpose dataset
  • Figure 5: Transparent objects used for robotic grasping experiments
  • ...and 1 more figures