Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ExoGS: A 4D Real-to-Sim-to-Real Framework for Scalable Manipulation Data Collection

Yiming Wang, Ruogu Zhang, Minyang Li, Hao Shi, Junbo Wang, Deyi Li, Jieji Ren, Wenhai Liu, Weiming Wang, Hao-Shu Fang

TL;DR

ExoGS tackles the data bottleneck in manipulation learning by delivering a robot-free 4D Real-to-Sim-to-Real framework that captures real-world manipulation sequences and reconstructs them as editable 3D Gaussian Splatting assets. It combines AirExo-3 for kinematically faithful demonstrations with a 3DGS-based replay and augmentation pipeline, enabling scalable, geometry-consistent data generation. A lightweight Mask Adapter injects instance-level semantic cues into ViT-based policies to reduce visual domain gaps and improve cross-scene robustness. Real-world experiments show improved data collection efficiency and policy generalization, with augmentation strategies and Mask Adapter delivering notable gains, though challenges remain for highly dynamic or severely varied backgrounds and deformable objects.

Abstract

Real-to-Sim-to-Real technique is gaining increasing interest for robotic manipulation, as it can generate scalable data in simulation while having narrower sim-to-real gap. However, previous methods mainly focused on environment-level visual real-to-sim transfer, ignoring the transfer of interactions, which could be challenging and inefficient to obtain purely in simulation especially for contact-rich tasks. We propose ExoGS, a robot-free 4D Real-to-Sim-to-Real framework that captures both static environments and dynamic interactions in the real world and transfers them seamlessly to a simulated environment. It provides a new solution for scalable manipulation data collection and policy learning. ExoGS employs a self-designed robot-isomorphic passive exoskeleton AirExo-3 to capture kinematically consistent trajectories with millimeter-level accuracy and synchronized RGB observations during direct human demonstrations. The robot, objects, and environment are reconstructed as editable 3D Gaussian Splatting assets, enabling geometry-consistent replay and large-scale data augmentation. Additionally, a lightweight Mask Adapter injects instance-level semantics into the policy to enhance robustness under visual domain shifts. Real-world experiments demonstrate that ExoGS significantly improves data efficiency and policy generalization compared to teleoperation-based baselines. Code and hardware files have been released on https://github.com/zaixiabalala/ExoGS.

ExoGS: A 4D Real-to-Sim-to-Real Framework for Scalable Manipulation Data Collection

TL;DR

ExoGS tackles the data bottleneck in manipulation learning by delivering a robot-free 4D Real-to-Sim-to-Real framework that captures real-world manipulation sequences and reconstructs them as editable 3D Gaussian Splatting assets. It combines AirExo-3 for kinematically faithful demonstrations with a 3DGS-based replay and augmentation pipeline, enabling scalable, geometry-consistent data generation. A lightweight Mask Adapter injects instance-level semantic cues into ViT-based policies to reduce visual domain gaps and improve cross-scene robustness. Real-world experiments show improved data collection efficiency and policy generalization, with augmentation strategies and Mask Adapter delivering notable gains, though challenges remain for highly dynamic or severely varied backgrounds and deformable objects.

Abstract

Real-to-Sim-to-Real technique is gaining increasing interest for robotic manipulation, as it can generate scalable data in simulation while having narrower sim-to-real gap. However, previous methods mainly focused on environment-level visual real-to-sim transfer, ignoring the transfer of interactions, which could be challenging and inefficient to obtain purely in simulation especially for contact-rich tasks. We propose ExoGS, a robot-free 4D Real-to-Sim-to-Real framework that captures both static environments and dynamic interactions in the real world and transfers them seamlessly to a simulated environment. It provides a new solution for scalable manipulation data collection and policy learning. ExoGS employs a self-designed robot-isomorphic passive exoskeleton AirExo-3 to capture kinematically consistent trajectories with millimeter-level accuracy and synchronized RGB observations during direct human demonstrations. The robot, objects, and environment are reconstructed as editable 3D Gaussian Splatting assets, enabling geometry-consistent replay and large-scale data augmentation. Additionally, a lightweight Mask Adapter injects instance-level semantics into the policy to enhance robustness under visual domain shifts. Real-world experiments demonstrate that ExoGS significantly improves data efficiency and policy generalization compared to teleoperation-based baselines. Code and hardware files have been released on https://github.com/zaixiabalala/ExoGS.
Paper Structure (23 sections, 9 equations, 9 figures, 2 tables)

This paper contains 23 sections, 9 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Sim-to-Real Transfer and 3D Gaussian Splatting
  4. Demonstration Data Collection
  5. Method
  6. Hardware Design: AirExo-3
  7. 4D Data Generation and Augmentation
  8. Manipulation Demonstration Collection with AirExo-3
  9. Digital Asset Generation via Multi-View Reconstruction
  10. Object Pose Estimation and Trajectory Processing
  11. Data Augmentation under Gaussian Rendering
  12. Policy Module: Mask Adapter
  13. Introduction
  14. Mask Head
  15. Mask-guided Token Modeling
  16. ...and 8 more sections

Figures (9)

  • Figure 1: Mechanical structure of the proposed data acquisition device. (a) Structural design of an individual joint module. (b) Overall structure of AirExo-3, consisting of seven articulated joints and a parallel gripper. (c) The target robotic platform used in this work.
  • Figure 2: Overview of the pipeline for reconstructing manipulation demonstrations collected with AirExo-3 using 3D Gaussian representations. Camera-based object pose tracking and joint angle encoding from AirExo-3 are combined to generate robot motions, which, together with the reconstructed digital assets, allow the full manipulation process to be faithfully replayed in simulation.
  • Figure 3: Overview of the proposed Mask Adapter. The module is trained in two stages. Stage 1 performs semantic segmentation pretraining using pixel-level supervision generated by the 3D Gaussian Splatting pipeline, yielding stable patch-level semantic labels for the background, robotic arm, and manipulated objects. Stage 2 incorporates these semantic cues into a ViT-based imitation learning policy via enhanced positional encodings and mask-guided attention, encouraging interaction-relevant token communication to improve robustness and cross-scene generalization under visual domain shifts.
  • Figure 4: Real-world experimental setup and task illustration. A Flexiv Rizon 4s robotic arm is used together with two eye-on-base Intel RealSense D415 cameras. During standard experiments and teleoperation data collection, only the left camera is used, while the other camera is reserved exclusively for evaluating camera viewpoint variations. Three manipulation tasks are designed in this work, whose detailed descriptions are provided in Sec. \ref{['sec:setup']}.
  • Figure 5: Task completion time comparison between AirExo-3 and teleoperation. Bars show the average over all volunteers, and colored dots denote individual averages, computed using successful trials only.
  • ...and 4 more figures