Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReFlow6D: Refraction-Guided Transparent Object 6D Pose Estimation via Intermediate Representation Learning

Hrishikesh Gupta, Stefan Thalhammer, Jean-Baptiste Weibel, Alexander Haberl, Markus Vincze

TL;DR

This work tackles the challenging problem of monocular 6D pose estimation for transparent objects, whose appearance is strongly affected by light refraction. It introduces ReFlow6D, which learns a refractive-intermediate representation comprising refractive flow, attenuation, surface-region attention, and a visibility mask to encode environment-invariant light deformation. A Patch-PnP-based regressor uses these features for direct 6D pose estimation, augmented by a novel object compositing loss to refine the intermediate representations. Empirical results on TOD and Trans6D-32K show state-of-the-art accuracy, and real-world grasping experiments demonstrate practical applicability, underscoring the advantage of refractive-based features over traditional geometric or edge cues.

Abstract

Transparent objects are ubiquitous in daily life, making their perception and robotics manipulation important. However, they present a major challenge due to their distinct refractive and reflective properties when it comes to accurately estimating the 6D pose. To solve this, we present ReFlow6D, a novel method for transparent object 6D pose estimation that harnesses the refractive-intermediate representation. Unlike conventional approaches, our method leverages a feature space impervious to changes in RGB image space and independent of depth information. Drawing inspiration from image matting, we model the deformation of the light path through transparent objects, yielding a unique object-specific intermediate representation guided by light refraction that is independent of the environment in which objects are observed. By integrating these intermediate features into the pose estimation network, we show that ReFlow6D achieves precise 6D pose estimation of transparent objects, using only RGB images as input. Our method further introduces a novel transparent object compositing loss, fostering the generation of superior refractive-intermediate features. Empirical evaluations show that our approach significantly outperforms state-of-the-art methods on TOD and Trans32K-6D datasets. Robot grasping experiments further demonstrate that ReFlow6D's pose estimation accuracy effectively translates to real-world robotics task. The source code is available at: https://github.com/StoicGilgamesh/ReFlow6D and https://github.com/StoicGilgamesh/matting_rendering.

ReFlow6D: Refraction-Guided Transparent Object 6D Pose Estimation via Intermediate Representation Learning

TL;DR

This work tackles the challenging problem of monocular 6D pose estimation for transparent objects, whose appearance is strongly affected by light refraction. It introduces ReFlow6D, which learns a refractive-intermediate representation comprising refractive flow, attenuation, surface-region attention, and a visibility mask to encode environment-invariant light deformation. A Patch-PnP-based regressor uses these features for direct 6D pose estimation, augmented by a novel object compositing loss to refine the intermediate representations. Empirical results on TOD and Trans6D-32K show state-of-the-art accuracy, and real-world grasping experiments demonstrate practical applicability, underscoring the advantage of refractive-based features over traditional geometric or edge cues.

Abstract

Transparent objects are ubiquitous in daily life, making their perception and robotics manipulation important. However, they present a major challenge due to their distinct refractive and reflective properties when it comes to accurately estimating the 6D pose. To solve this, we present ReFlow6D, a novel method for transparent object 6D pose estimation that harnesses the refractive-intermediate representation. Unlike conventional approaches, our method leverages a feature space impervious to changes in RGB image space and independent of depth information. Drawing inspiration from image matting, we model the deformation of the light path through transparent objects, yielding a unique object-specific intermediate representation guided by light refraction that is independent of the environment in which objects are observed. By integrating these intermediate features into the pose estimation network, we show that ReFlow6D achieves precise 6D pose estimation of transparent objects, using only RGB images as input. Our method further introduces a novel transparent object compositing loss, fostering the generation of superior refractive-intermediate features. Empirical evaluations show that our approach significantly outperforms state-of-the-art methods on TOD and Trans32K-6D datasets. Robot grasping experiments further demonstrate that ReFlow6D's pose estimation accuracy effectively translates to real-world robotics task. The source code is available at: https://github.com/StoicGilgamesh/ReFlow6D and https://github.com/StoicGilgamesh/matting_rendering.
Paper Structure (14 sections, 8 equations, 5 figures, 4 tables)

This paper contains 14 sections, 8 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Transparent object perception and manipulation
  4. Transparent object 6D pose estimation and image matting
  5. ReFlow6D
  6. Refractive-intermediate representation
  7. Loss Functions
  8. Experiments
  9. Experimental Setup
  10. Evaluation
  11. Comparison with the benchmark
  12. Ablation Studies
  13. Real-world robot experiments
  14. Conclusion

Figures (5)

  • Figure 1: Intermediate representation for pose estimation: The figure shows the effectiveness of the refractive-intermediate representation vs Geometric and edge intermediate layers applied to 6D pose estimation. The green 3D Bbox shows groundtruth, while the blue shows the estimation.
  • Figure 2: Framework of ReFlow6D: (a) Given an RGB image $I$ we use off-the-shelf object detector for detecting transparent objects. (b) The RFA feature regression network takes then zoomed-in RoI as input and predicts several refractive-intermediate representation. (c) These intermediate features are then concatenated and provided as input to the Patch-PnP. (d) The Patch-PnP directly regresses the 6D object pose of the transparent object.
  • Figure 3: Transparent object Compositing: Examples of transparent object compositing from the TOD and Trans32K-6D datasets on random COCO backgrounds. Used for additional supervision loss for refining the estimated RFA.
  • Figure 4: Qualitative Results of ReFlow6D: (a) Qualitative results on the TOD dataset. (b) Qualitative results on the Trans32K-6D dataset. Estimates are shown in cropped images for visibility. No estimates are shown for the TGF-Net method as the authors did not publish their code.
  • Figure 5: Grasping qualitative results: On the left we show (a) Examples of all three different scenarios. (b) Grasping example of the object "Canister" for all 3 scenarios. (c) Grasping example of the object "SmallBottle" for all 3 scenarios. (d) Grasping of the object "LargeBottle" for all 3 scenarios. On the right we show four cases of illumination (a) Light sifting through the semi-permeable blind covering the only window of the room. (b) Artificial ambient light. (c) Natural light. While the fourth case of illumination i.e, superposition of artificial ambient and natural light is shown on the left side of images with the grasping senarios.