Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MonoDiff9D: Monocular Category-Level 9D Object Pose Estimation via Diffusion Model

Jian Liu, Wei Sun, Hui Yang, Jin Zheng, Zichen Geng, Hossein Rahmani, Ajmal Mian

TL;DR

This work tackles monocular category-level 9D object pose estimation without relying on CAD models or depth sensors. It introduces MonoDiff9D, which uses zero-shot coarse depth from DINOv2 to form a conditional point cloud and employs an LVM-guided encoding to condition a transformer-based diffusion denoiser, enabling end-to-end differentiable pose recovery. The approach achieves state-of-the-art results on CAMERA25 and REAL275 without shape priors, demonstrating robust performance and generalization in the wild with near real-time inference. By eliminating depth sensors and CAD priors, MonoDiff9D offers a practical pathway for accurate 9D pose estimation in monocular, resource-constrained robotic systems.

Abstract

Object pose estimation is a core means for robots to understand and interact with their environment. For this task, monocular category-level methods are attractive as they require only a single RGB camera. However, current methods rely on shape priors or CAD models of the intra-class known objects. We propose a diffusion-based monocular category-level 9D object pose generation method, MonoDiff9D. Our motivation is to leverage the probabilistic nature of diffusion models to alleviate the need for shape priors, CAD models, or depth sensors for intra-class unknown object pose estimation. We first estimate coarse depth via DINOv2 from the monocular image in a zero-shot manner and convert it into a point cloud. We then fuse the global features of the point cloud with the input image and use the fused features along with the encoded time step to condition MonoDiff9D. Finally, we design a transformer-based denoiser to recover the object pose from Gaussian noise. Extensive experiments on two popular benchmark datasets show that MonoDiff9D achieves state-of-the-art monocular category-level 9D object pose estimation accuracy without the need for shape priors or CAD models at any stage. Our code will be made public at https://github.com/CNJianLiu/MonoDiff9D.

MonoDiff9D: Monocular Category-Level 9D Object Pose Estimation via Diffusion Model

TL;DR

This work tackles monocular category-level 9D object pose estimation without relying on CAD models or depth sensors. It introduces MonoDiff9D, which uses zero-shot coarse depth from DINOv2 to form a conditional point cloud and employs an LVM-guided encoding to condition a transformer-based diffusion denoiser, enabling end-to-end differentiable pose recovery. The approach achieves state-of-the-art results on CAMERA25 and REAL275 without shape priors, demonstrating robust performance and generalization in the wild with near real-time inference. By eliminating depth sensors and CAD priors, MonoDiff9D offers a practical pathway for accurate 9D pose estimation in monocular, resource-constrained robotic systems.

Abstract

Object pose estimation is a core means for robots to understand and interact with their environment. For this task, monocular category-level methods are attractive as they require only a single RGB camera. However, current methods rely on shape priors or CAD models of the intra-class known objects. We propose a diffusion-based monocular category-level 9D object pose generation method, MonoDiff9D. Our motivation is to leverage the probabilistic nature of diffusion models to alleviate the need for shape priors, CAD models, or depth sensors for intra-class unknown object pose estimation. We first estimate coarse depth via DINOv2 from the monocular image in a zero-shot manner and convert it into a point cloud. We then fuse the global features of the point cloud with the input image and use the fused features along with the encoded time step to condition MonoDiff9D. Finally, we design a transformer-based denoiser to recover the object pose from Gaussian noise. Extensive experiments on two popular benchmark datasets show that MonoDiff9D achieves state-of-the-art monocular category-level 9D object pose estimation accuracy without the need for shape priors or CAD models at any stage. Our code will be made public at https://github.com/CNJianLiu/MonoDiff9D.

Paper Structure

This paper contains 23 sections, 9 equations, 3 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. RGBD-Based and Depth-Based Methods
  4. RGB-Based Methods
  5. WORKFLOW of MONODIFF9D
  6. Point Cloud Restoration
  7. Conditional Encoding
  8. Pose Diffusion
  9. Forward Diffusion
  10. Reverse Diffusion
  11. Training Protocol
  12. EXPERIMENTS
  13. Evaluation Datasets and Metrics
  14. Implementation Details
  15. Comparisons with State-of-the-Art Methods
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Comparison between (a) shape prior-based RGBD methods and (b) the proposed MonoDiff9D. (a) leverages shape prior, RGB image, and point cloud to reconstruct the NOCS representation of the object, and uses NOCS shape alignment to solve the object pose. (b) first estimates the coarse object point cloud based on DINOv2 DINOv2, and then uses the RGB image and the point cloud as conditions to guide the diffusion model for recovering the object pose.
  • Figure 2: Overall workflow of the proposed MonoDiff9D, which includes three parts: point cloud restoration, conditional encoding, and pose diffusion.
  • Figure 3: Qualitative results comparison with SOTA method DMSR DMSR on the CAMERA25 (left) and REAL275 (right) datasets. The blue arrows point to areas where DMSR has significantly higher error than our method.