IRIS: An Immersive Robot Interaction System

TL;DR

IRIS tackles reproducibility and reusability challenges in XR-based robot data collection by introducing a general, extensible framework that bridges multiple simulators and real-world environments. The core ideas are a Unified Scene Representation (USR) that serializes scene geometry, materials, and textures for cross-simulator rendering, coupled with cross-scene, cross-embodiment, cross-simulator, cross-reality, cross-platform, and cross-user capabilities. Key contributions include deformable object manipulation in XR, multi-user collaboration, and a ROS-friendly yet lightweight communication design that decouples XR from rigid simulator dependencies. Experimental results show efficient data collection and competitive policy training in both simulation and real-world tasks, with tangible improvements over traditional Tele-Op baselines. Overall, IRIS offers a practical, open-source pipeline for scalable, immersive robot data collection across diverse hardware and software stacks.

Abstract

This paper introduces IRIS, an Immersive Robot Interaction System leveraging Extended Reality (XR). Existing XR-based systems enable efficient data collection but are often challenging to reproduce and reuse due to their specificity to particular robots, objects, simulators, and environments. IRIS addresses these issues by supporting immersive interaction and data collection across diverse simulators and real-world scenarios. It visualizes arbitrary rigid and deformable objects, robots from simulation, and integrates real-time sensor-generated point clouds for real-world applications. Additionally, IRIS enhances collaborative capabilities by enabling multiple users to simultaneously interact within the same virtual scene. Extensive experiments demonstrate that IRIS offers efficient and intuitive data collection in both simulated and real-world settings.

Figures (19)

• Figure 1: We present IRIS, an Immersive Robot Interaction System designed to support various simulators and real-world scenarios.
• Figure 2: Paradigms of the system architecture in both simulation (left) and real world (right). All the devices are connected through a Wi-Fi router. In the left image, the simulation updates the scene to all headsets using the SimPublisher. A spatial anchor is used to align the virtual scenes across different headsets. In the right image, a sensor generates a point cloud transmitted to the XR headset, allowing the operator to clearly observe the manipulated object in front of the follower robot.
• Figure 3: Collaborative manipulation in simulation via IRIS. The left image shows the collaborative manipulation for handing over a hammer between two Franka Panda robots by KT, and the right image shows that collaborative manipulation for handing over a red board between two Aloha 2 Arms by MC.
• Figure 4: Playing table tennis with RL agent in Fancy Gym environment. The RL agent policy is trained with Deep Black-Box Reinforcement Learning (BBRL) otto2023deepotto2023mp3
• Figure 5: IRIS Real-world Application. This setup features two Franka robots: a leader robot controlled by a user wearing a Meta Quest 3 headset and a follower robot that mirrors its movements. A depth camera captures the environment for real-time point cloud visualization in XR.