RoboKeyGen: Robot Pose and Joint Angles Estimation via Diffusion-based 3D Keypoint Generation

TL;DR

A novel framework that bifurcates the high-dimensional prediction task into two manageable subtasks: 2D keypoints detection and lifting 2D keypoints to 3D, and outperforms the state-of-the-art render&compare method and achieves higher inference speed.

Abstract

Estimating robot pose and joint angles is significant in advanced robotics, enabling applications like robot collaboration and online hand-eye calibration.However, the introduction of unknown joint angles makes prediction more complex than simple robot pose estimation, due to its higher dimensionality.Previous methods either regress 3D keypoints directly or utilise a render&compare strategy. These approaches often falter in terms of performance or efficiency and grapple with the cross-camera gap problem.This paper presents a novel framework that bifurcates the high-dimensional prediction task into two manageable subtasks: 2D keypoints detection and lifting 2D keypoints to 3D. This separation promises enhanced performance without sacrificing the efficiency innate to keypoint-based techniques.A vital component of our method is the lifting of 2D keypoints to 3D keypoints. Common deterministic regression methods may falter when faced with uncertainties from 2D detection errors or self-occlusions.Leveraging the robust modeling potential of diffusion models, we reframe this issue as a conditional 3D keypoints generation task. To bolster cross-camera adaptability, we introduce theNormalised Camera Coordinate Space (NCCS), ensuring alignment of estimated 2D keypoints across varying camera intrinsics.Experimental results demonstrate that the proposed method outperforms the state-of-the-art render\&compare method and achieves higher inference speed.Furthermore, the tests accentuate our method's robust cross-camera generalisation capabilities.We intend to release both the dataset and code in https://nimolty.github.io/Robokeygen/

Figures (4)

• Figure 1: RoboKeyGen. Given RGB images, we aim to estimate the robot pose and joint angles. We achieve this goal by decoupling it into two more tractable tasks: 2D keypoints detection and lifting 2D keypoints to 3D.
• Figure 2: The inference pipeline of RoboKeyGen. (A) Combined with the RGB image $I$, predicted segmentation mask and positional embedding prior $\mathcal{F}$, we firstly predict 2D keypoints $c$ through the detection network $\Psi_{\omega}$. (B) Conditioning on 2D detections, we generate 3D $X^{cam}$ via the score network $\Phi_{\zeta}$. (C) Finally, we predict joint angles from $X^{cam}$ and recover $X^{rob}$ based on URDF files. We do pose fitting between $X^{cam}$ and $X^{rob}$ to acquire the robot pose.
• Figure 3: Visualisation results on real-world datasets. Green edges are ground truth while red edges are rendered via estimated robot pose and joint angles. White boxes highlight regions where ours (online) performs better than RoboPose (online) Labbe2021SingleviewRP.
• Figure 4: Ablation on different number of 3D keypoints candidates $K$. We finally adopt $K=10$ in implementation.