Learning to Estimate the Pose of a Peer Robot in a Camera Image by Predicting the States of its LEDs

TL;DR

This work tackles estimating the relative 6D pose of a peer robot from a monocular image under scarce pose-label data. It introduces a fully convolutional network trained with a self-supervised LED-based pretext task that predicts image-space position, distance, orientation, and independent LED states, using an attention-like weighting $ ext{wm}_{m{ ilde{P}}}$ to connect maps to scalars. The approach enables learning useful representations on unlabeled data and transferring them to the downstream pose task with few labels, showing significant improvements over baselines and CLIP features, and generalizing to unseen environments. The findings demonstrate practical impact by reducing reliance on external tracking infrastructure and enabling scalable, robust multi-robot pose estimation in varied settings.

Abstract

We consider the problem of training a fully convolutional network to estimate the relative 6D pose of a robot given a camera image, when the robot is equipped with independent controllable LEDs placed in different parts of its body. The training data is composed by few (or zero) images labeled with a ground truth relative pose and many images labeled only with the true state (\textsc{on} or \textsc{off}) of each of the peer LEDs. The former data is expensive to acquire, requiring external infrastructure for tracking the two robots; the latter is cheap as it can be acquired by two unsupervised robots moving randomly and toggling their LEDs while sharing the true LED states via radio. Training with the latter dataset on estimating the LEDs' state of the peer robot (\emph{pretext task}) promotes learning the relative localization task (\emph{end task}). Experiments on real-world data acquired by two autonomous wheeled robots show that a model trained only on the pretext task successfully learns to localize a peer robot on the image plane; fine-tuning such model on the end task with few labeled images yields statistically significant improvements in 6D relative pose estimation with respect to baselines that do not use pretext-task pre-training, and alternative approaches. Estimating the state of multiple independent LEDs promotes learning to estimate relative heading. The approach works even when a large fraction of training images do not include the peer robot and generalizes well to unseen environments.

Figures (5)

• Figure 1: A DJI RoboMaster S1 visually estimating the relative pose of a peer robot, using a model trained on an autonomously-collected dataset in which the four LED states of its peer robot are known for all images; instead, the true relative pose of its peer is known in few, or even zero images.
• Figure 2: Our fully convolutional network model takes an image as input and predicts maps for the robot's image space position $\bm{\hat{P}}$, distance $\bm{\hat{D}}$, orientation $\bm{\hat{O}}$ and state of the LEDs $\bm{\hat{L}}_i$. We obtain scalar values from the maps by element-wise multiplication (denoted as $\circ$) with $norm(\bm{\hat{P}})$, acting as an attention mechanism. By optimizing $\hat{l}_i$, the model learns to estimate the robot's LED state and position in the image; gradients for $\bm{\hat{P}}$ resulting from the optimization of $\hat{d}$ and $\hat{\psi}$ are blocked (see Section \ref{['sec:model:losses']} for details).
• Figure 3: Six samples from the unlabeled training set $\mathcal{T}_u^a$, where only 22% of images feature a visible robot; ground truth robot poses are depicted with a blue bounding box.
• Figure 4: Predicted LED maps by the pretext model trained on random 3.5k samples from $\mathcal{T}_u^\nu$. From left to right, input image, left LED map, and right LED map. Top image depicts the robot with left LED visible and turned on; bottom image depicts right LED visible and turned off. Position map cells with low probability of having a robot are depicted in gray. Predicted LED state scalars are reported on the top right corners of the LED maps.
• Figure 5: Six images from unseen environments and predicted robot's bounding boxes (red) generated by the downstream model trained on $\mathcal{T}_\ell^{1000}$.