Fiducial Exoskeletons: Image-Centric Robot State Estimation

TL;DR

The paper tackles the challenge of accurate 3D robot state estimation and precise control on low-cost arms by removing reliance on high-precision actuators and iterative calibration. It introduces Fiducial Exoskeletons (FidEx), a vision-centric framework that estimates per-link 6D poses from a single RGB image, then recovers the full joint state, camera extrinsics, and calibration through a lightweight optimization; fiducial markers attached to each link provide robust, marker-based pose measurements. A simple state-estimation refinement control loop further improves motion accuracy. Across experiments on a low-cost arm, FidEx achieves substantial reductions in end-effector pose error (around 75%) and control error (around 45%), demonstrates robustness to marker occlusion and unfavorable orientations, and runs significantly faster than differentiable-rendering baselines, highlighting practical impact for affordable robotics and real-time 3D control.

Abstract

We introduce Fiducial Exoskeletons, an image-based reformulation of 3D robot state estimation that replaces cumbersome procedures and motor-centric pipelines with single-image inference. Traditional approaches - especially robot-camera extrinsic estimation - often rely on high-precision actuators and require time-consuming routines such as hand-eye calibration. In contrast, modern learning-based robot control is increasingly trained and deployed from RGB observations on lower-cost hardware. Our key insight is twofold. First, we cast robot state estimation as 6D pose estimation of each link from a single RGB image: the robot-camera base transform is obtained directly as the estimated base-link pose, and the joint state is recovered via a lightweight global optimization that enforces kinematic consistency with the observed link poses (optionally warm-started with encoder readings). Second, we make per-link 6D pose estimation robust and simple - even without learning - by introducing the fiducial exoskeleton: a lightweight 3D-printed mount with a fiducial marker on each link and known marker-link geometry. This design yields robust camera-robot extrinsics, per-link SE(3) poses, and joint-angle state from a single image, enabling robust state estimation even on unplugged robots. Demonstrated on a low-cost robot arm, fiducial exoskeletons substantially simplify setup while improving calibration, state accuracy, and downstream 3D control performance. We release code and printable hardware designs to enable further algorithm-hardware co-design.

Figures (10)

• Figure 1: FidEx Overview. After installing a lightweight 3D-printed fiducial exoskeleton --- one marker-equipped piece per link --- FidEx enables instant 3D state estimation from a single RGB image. Given one view, we recover the external camera pose (robot base pose), the full joint state by optimizing over detected 6D link poses, and the joint calibration by aligning optimized joint angles with raw motor readings.
• Figure 2: Overview of using Fiducial Exoskeletons for robot state estimation. (1a, top-left) Each link is fitted with a fiducial exoskeleton, a 3D-printed mount with a flat marker plane and a known marker-to-link coordinate transformation, enabling single-image 6D pose estimation for each link without any iterative calibration (Section \ref{['text:fiducial_exo']}). (1b, top-right) The exoskeleton for each link is printed and mounted on the robot. (2, bottom) From an RGB image, we estimate all link poses, also immediately yielding the camera pose directly from the base's marker [2a]. A fast optimization recovers the joint angles [2b] which best match the observed link poses (Section \ref{['text:state_recovery']}). We also recover the robot's calibration offset [2c] by comparing the optimized joints to the raw encoder joints. The final joint estimate tightly matches the physical robot, observed by the rendered robot overlay (rightmost).
• Figure 3: The pseudo-code for our visual state-estimating control. Top: the pseudo-code of our control loop, where we first move the arm naively to the target position, estimate the current robot state, move the delta between the observed robot and target robot state, and finally re-calibrate the robot offsets for the next target motion. Bottom: Illustrated steps of the control loop with the target state (Left), then the naive motion execution (Middle), and lastly the state-based refinement to better match the target state. We also highlight insets on the end-effector to better emphasize the differences between the target and physical states.
• Figure 4: Comparison to Dr. Robot on robot-camera pose and joint state estimation. We compare our (left) robot-camera pose and joint state estimates to those from Dr. Robot liu2024differentiablerobotrendering (right), a rendering-based robot state estimation method, visualizing the input image (top row) and the re-rendered image (bottom row) using the inferred joints and camera pose. Dr. Robot's rendering is visualized using their internal splatting renderer and ours using the simulator renderer with our inferred parameters. Dr. Robot infers pose and joints parameters using differentiable rendering, which is known to be sensitive to local minima and initialization; its recovered pose is frequently misaligned with the input whereas our inferred parameters align near pixel-perfect with the input.
• Figure 5: State estimation results. (Left) Using raw encoder readings for state estimation (left), the rendered-robot is not well aligned with the physical robot. Using the fiducial exoskeletons, even without using any encoder readings as input, the robot re-render is aligned with the physical robot (Right). For each method, we plot the full-image overlay (inner left) and a highlighted inset on the end-effector (inner right).