Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Depth Restoration of Hand-Held Transparent Objects for Human-to-Robot Handover

Ran Yu, Haixin Yu, Shoujie Li, Huang Yan, Ziwu Song, Wenbo Ding

TL;DR

A real-world human-to-robot handover system based on HADR, demonstrating its potential in human-robot interaction applications and better performance and generalization ability compared with existing methods.

Abstract

Transparent objects are common in daily life, while their optical properties pose challenges for RGB-D cameras to capture accurate depth information. This issue is further amplified when these objects are hand-held, as hand occlusions further complicate depth estimation. For assistant robots, however, accurately perceiving hand-held transparent objects is critical to effective human-robot interaction. This paper presents a Hand-Aware Depth Restoration (HADR) method based on creating an implicit neural representation function from a single RGB-D image. The proposed method utilizes hand posture as an important guidance to leverage semantic and geometric information of hand-object interaction. To train and evaluate the proposed method, we create a high-fidelity synthetic dataset named TransHand-14K with a real-to-sim data generation scheme. Experiments show that our method has better performance and generalization ability compared with existing methods. We further develop a real-world human-to-robot handover system based on HADR, demonstrating its potential in human-robot interaction applications.

Depth Restoration of Hand-Held Transparent Objects for Human-to-Robot Handover

TL;DR

A real-world human-to-robot handover system based on HADR, demonstrating its potential in human-robot interaction applications and better performance and generalization ability compared with existing methods.

Abstract

Transparent objects are common in daily life, while their optical properties pose challenges for RGB-D cameras to capture accurate depth information. This issue is further amplified when these objects are hand-held, as hand occlusions further complicate depth estimation. For assistant robots, however, accurately perceiving hand-held transparent objects is critical to effective human-robot interaction. This paper presents a Hand-Aware Depth Restoration (HADR) method based on creating an implicit neural representation function from a single RGB-D image. The proposed method utilizes hand posture as an important guidance to leverage semantic and geometric information of hand-object interaction. To train and evaluate the proposed method, we create a high-fidelity synthetic dataset named TransHand-14K with a real-to-sim data generation scheme. Experiments show that our method has better performance and generalization ability compared with existing methods. We further develop a real-world human-to-robot handover system based on HADR, demonstrating its potential in human-robot interaction applications.
Paper Structure (12 sections, 5 equations, 7 figures, 3 tables)

This paper contains 12 sections, 5 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Depth Restoration of Transparent Objects
  4. Datasets for Transparent Objects Perception
  5. Methodology
  6. TransHand-14K Dataset Generation
  7. Hand-Aware Depth Restoration
  8. H2R Handover Workflow of Transparent Objects
  9. Experiments
  10. Depth Restoration Experiments on TransHand-14K
  11. Real-World Experiments
  12. Conclusion

Figures (7)

  • Figure 1: RGB-D sensors meet challenges in estimating the depth of transparent objects, especially hand-held objects. We present a hand-aware depth restoration method that reconstructs the corrupted point cloud with the guidance of hand pose information. Our method can be used for human-to-robot handover, highlighting its application value.
  • Figure 2: TransHand-14K data generation method. (a) A real-to-sim hand pose generation scheme is introduced. (b) We use a MANUS VR glove to capture the pose data. (c) Eight transparent objects are included in TransHand-14K.
  • Figure 3: Visualization of proposed dataset TransHand-14K.
  • Figure 4: Overview of HADR. (a) Ray and voxel features are generated from the RGB and corrupted point cloud. (b) We introduce the handhold pose as an important guidance for depth restoration. This feature is related to geometric and semantic information of hand-object interaction. (c) The terminated probability and position of each ray-voxel pair are predicted. The final restored depth is obtained by ray-wise maxpooling.
  • Figure 5: Overview of the proposed handover workflow. The whole handover process is divided into three stages: 1) Wait & Observe, 2) Approach & React, and 3) Grasp & Retrieve.
  • ...and 2 more figures