GeCo-SRT: Geometry-aware Continual Adaptation for Robotic Cross-Task Sim-to-Real Transfer

TL;DR

GeCo-SRT, a geometry-aware continual adaptation method, utilizes domain-invariant and task-invariant knowledge from local geometric features as a transferable foundation to accelerate adaptation during subsequent sim-to-real transfers, thereby enabling effective and efficient adaptation to novel tasks.

Abstract

Bridging the sim-to-real gap is important for applying low-cost simulation data to real-world robotic systems. However, previous methods are severely limited by treating each transfer as an isolated endeavor, demanding repeated, costly tuning and wasting prior transfer experience. To move beyond isolated sim-to-real, we build a continual cross-task sim-to-real transfer paradigm centered on knowledge accumulation across iterative transfers, thereby enabling effective and efficient adaptation to novel tasks. Thus, we propose GeCo-SRT, a geometry-aware continual adaptation method. It utilizes domain-invariant and task-invariant knowledge from local geometric features as a transferable foundation to accelerate adaptation during subsequent sim-to-real transfers. This method starts with a geometry-aware mixture-of-experts module, which dynamically activates experts to specialize in distinct geometric knowledge to bridge observation sim-to-real gap. Further, the geometry-expert-guided prioritized experience replay module preferentially samples from underutilized experts, refreshing specialized knowledge to combat forgetting and maintain robust cross-task performance. Leveraging knowledge accumulated during iterative transfer, GeCo-SRT method not only achieves 52% average performance improvement over the baseline, but also demonstrates significant data efficiency for new task adaptation with only 1/6 data. We hope this work inspires approaches for efficient, low-cost cross-task sim-to-real transfer.

Figures (9)

• Figure 1: Our GeCo-SRT method accumulates transferable geometric knowledge from sequential sim-to-real tasks (Task $1...N$), which facilitates effective and efficient transfer to novel tasks.
• Figure 2: Overview of our geometry-aware continual adaptation method. (a) demonstrates the sim-to-real transfer with human correction pipeline, which translates sim-to-real heuristics into learnable human correction trajectories. (b) represents the Geometry-aware Mixture-of-Experts module, which extracts local geometric features to dynamically activate experts, enabling them to specialize in distinct geometric knowledge for effective sim-to-real transfer. (c) illustrates the Geometry-expert-guided Prioritized Experience Replay, which leverages expert utilization to prioritize historical data, thereby mitigating catastrophic forgetting and promoting knowledge reuse.
• Figure 3: We demonstrate the task setup and real-world setup. (a) illustrates 4 robotic manipulation task in simulation and real-world scenarios. (b) shows that the real-world setup, which consists of a XARM equipped with a robotiq-140 gripper.
• Figure 4: Data efficiency comparison of training "From Scratch" vs. "From Continued". The performance gain (hatched area) highlights our method's ability to utilize transferable geometric knowledge for cross-task sim-to-real transfer, which enhances data efficiency when adapting to new tasks, especially in low-data regimes.
• Figure 5: Real-world workspace setup for human-in-the-loop data collection. The human operator provides online correction through a 3Dconnexion SpaceMouse (marked with a red box) while monitoring the robot’s execution.