Diffusing in Someone Else's Shoes: Robotic Perspective Taking with Diffusion

TL;DR

The paper tackles perspective-taking in robot imitation learning by translating third-person demonstrations into first-person robot views using a conditional diffusion model. It introduces a two-path U-Net–like architecture with latent condition connections, trains on three datasets (simulated NICOL and real-world EMIL and RH20T), and demonstrates superior first-person image generation compared to pix2pix and CycleGAN. Beyond image translation, it shows that conditioning on generated first-person views improves downstream joint-value estimation, highlighting practical benefits for data-efficient imitation learning. The work suggests diffusion-based perspective translation can unlock abundant third-person demonstrations for learning in robotics, with robust performance across diverse environments. The results also motivate richer evaluation metrics to capture qualitative improvements in high-level structure and pose fidelity.

Abstract

Humanoid robots can benefit from their similarity to the human shape by learning from humans. When humans teach other humans how to perform actions, they often demonstrate the actions, and the learning human imitates the demonstration to get an idea of how to perform the action. Being able to mentally transfer from a demonstration seen from a third-person perspective to how it should look from a first-person perspective is fundamental for this ability in humans. As this is a challenging task, it is often simplified for robots by creating demonstrations from the first-person perspective. Creating these demonstrations allows for an easier imitation but requires more effort. Therefore, we introduce a novel diffusion model that enables the robot to learn from the third-person demonstrations directly by learning to generate the first-person perspective from the third-person perspective. The model translates the size and rotations of objects and the environment between the two perspectives. This allows us to utilise the benefits of easy-to-produce third-person demonstrations and easy-to-imitate first-person demonstrations.

Figures (8)

• Figure 1: The results of our model for one example of each dataset. Top row is our dataset created with the simulated NICOL robot. An example of the EMIL dataset c28 which was recorded in a real-world scenario is in the second row. The third row is for the RH20T dataset c30 which was recorded in a more cluttered real-world scenario. The resolution is in accordance with the model's input and output.
• Figure 2: Our architecture has two inputs: the condition, which is the third-person perspective of the robot in the top middle and the noised output during training or, as seen in the top left image, simply noise during inference. Then, the top left image shows the noise that is the input during inference, and the top right image shows the output of our model, the first-person perspective from the robot. Both input images are encoded, each through one of the downward paths. The encoded version of the noise is directly led into the upward path that decodes the image to the output, while the encoded version of the condition is led into the upward path through latent connections.
• Figure 3: Qualitative comparisons between our model and pix2pix as well as CycleGAN. Five examples (A, B, C, D, E) are shown to illustrate the differences between the models. CycleGan has almost no variance between its output, pix2pix mostly seems to have the correct poses except for example C, but is often unable to correctly recreate the hands and finger structure. Our model has always the correct pose and fully recreates the hands. The resolution is in accordance with the model's input and output.
• Figure 4: One example of predicting the first-person perspective, where the direct prediction differs from the prediction after the iterative denoising. In the direct prediction, two thumbs can be identified on the hand, whereas only one is left after the iterative denoising, although in the wrong position. The resolution is in accordance with the model's input and output.
• Figure 5: Third-person perspective generation for the EMIL dataset. The resolution is in accordance with the model's input and output.