RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph

TL;DR

This work tackles monocular robot pose estimation by addressing data scarcity and 3D-2D integration. It introduces RoboTAG, a dual-branch Topological Alignment Graph that injects 3D priors via a forward-edge structure and enforces 2D-3D consistency through closed-loop supervision. The approach enables self-supervised training on unannotated in-the-wild data and achieves state-of-the-art performance across multiple robot types, while maintaining real-time inference. Overall, RoboTAG demonstrates that a carefully designed topological graph can leverage 3D priors and cross-modal supervision to alleviate labeling bottlenecks in robotic perception, with strong generalization to real-world data.

Abstract

Estimating robot pose from a monocular RGB image is a challenge in robotics and computer vision. Existing methods typically build networks on top of 2D visual backbones and depend heavily on labeled data for training, which is often scarce in real-world scenarios, causing a sim-to-real gap. Moreover, these approaches reduce the 3D-based problem to 2D domain, neglecting the 3D priors. To address these, we propose Robot Topological Alignment Graph (RoboTAG), which incorporates a 3D branch to inject 3D priors while enabling co-evolution of the 2D and 3D representations, alleviating the reliance on labels. Specifically, the RoboTAG consists of a 3D branch and a 2D branch, where nodes represent the states of the camera and robot system, and edges capture the dependencies between these variables or denote alignments between them. Closed loops are then defined in the graph, on which a consistency supervision across branches can be applied. This design allows us to utilize in-the-wild images as training data without annotations. Experimental results demonstrate that our method is effective across robot types, highlighting its potential to alleviate the data bottleneck in robotics.

Figures (4)

• Figure 1: The intuition of RoboTAG. (a) Existing works predict every state (square) of the camera and robot system independently, and apply supervision on each of them. (b) Our method constructs a topological graph, RoboTAG, by introducing a 3D branch (blue) with forward edges (solid arrows) and alignment edges (dotted arrows) connecting system states. Multiple forward edges to a state indicate it depends on several others. Closed loops are defined in this topology to enable 2D-3D consistency supervision.
• Figure 2: Overview of the proposed method. The framework consists of a 3D branch and a 2D branch, which are deeply intertwined as a topological graph. The nodes represent the states of the camera and robot system, and the edges represent dependencies between these variables (solid arrow) or denote alignments between equivalent nodes (dotted arrow). The closed loops in the topology enable 2D-3D consistency supervision, allowing the 2D and 3D branches to co-evolve. Gradients from 2D-3D alignment losses flow in the closed loops, enhancing the Neural Networks on the forward edges. Part of the graph, including the forward kinematics keypoints, is omitted for clarity.
• Figure 3: Qualitative results on the Panda real and synthetic datasets in Dream. The predicted robot pose is overlaid on top of the input image. The smaller the gap between the grey render and original robot is, the better the prediction is. Our method achieves the best performance for most robot parts.
• Figure 4: Ablation study for the effectiveness of 3D priors and TAG alignment on the Panda and Kuka datasets. Introducing 3D priors by adding 2D-3D feature fusion leads to a small performance gain of 0.45%, and TAG further brings a boost of 1.6%, demonstrating the effectiveness of our method.