Sim-and-Human Co-training for Data-Efficient and Generalizable Robotic Manipulation

TL;DR

SimHum addresses the data-efficiency and generalization gaps in robotic manipulation by jointly leveraging scalable simulation data and real-world human demonstrations. It introduces a two-stage diffusion-based policy with modular, source-dependent components that extract transferable kinematic priors from simulation and visual priors from human data, then fine-tunes on a small real-robot dataset. Empirical results show substantial improvements over real-only and single-source baselines, including up to $+35.0\%$ SR gains in OOD settings and a $7.1\times$ advantage over Real-only policies, using only $80$ real episodes. The approach offers a scalable pathway to data-efficient, generalizable robotic manipulation and informs future integration of heterogeneous data sources for robotic foundation models.

Abstract

Synthetic simulation data and real-world human data provide scalable alternatives to circumvent the prohibitive costs of robot data collection. However, these sources suffer from the sim-to-real visual gap and the human-to-robot embodiment gap, respectively, which limits the policy's generalization to real-world scenarios. In this work, we identify a natural yet underexplored complementarity between these sources: simulation offers the robot action that human data lacks, while human data provides the real-world observation that simulation struggles to render. Motivated by this insight, we present SimHum, a co-training framework to simultaneously extract kinematic prior from simulated robot actions and visual prior from real-world human observations. Based on the two complementary priors, we achieve data-efficient and generalizable robotic manipulation in real-world tasks. Empirically, SimHum outperforms the baseline by up to $\mathbf{40\%}$ under the same data collection budget, and achieves a $\mathbf{62.5\%}$ OOD success with only 80 real data, outperforming the real only baseline by $7.1\times$. Videos and additional information can be found at \href{https://kaipengfang.github.io/sim-and-human}{project website}.

Figures (17)

• Figure 1: Sim-and-Human Co-training.(a) Simulation data offers transferable actions but suffers from the Sim2Real gap; (b) human data provides realistic observations but is limited by the Human2Robot gap. (c) Our Sim-and-Human Co-training framework unifies transferable sim robot actions and human observations to mutually mitigate their respective gaps. (d) Consequently, our approach demonstrates exceptional data efficiency and generalization capabilities under OOD settings.
• Figure 2: Model Overview. Our approach consists of two main stages: (1) Sim-and-Human Pre-training that disentangles visual and action representations from simulation and human data to establish a generalized manipulation prior, and (2) Real-robot fine-tuning that couples the real-world adaptor with the action encoder-decoder to achieve data-efficient and generalizable real-world manipulation.
• Figure 3: Experimental setup for evaluating the impact of the background factor ($\mathcal{F}_{bg}$) in human data. We exclude $\mathcal{F}_{bg}$-related samples from the human data and design two corresponding OOD scenarios to quantify how this reduction in diversity affects the policy's ability to generalize to unseen backgrounds.
• Figure 4: Human data enhances visual generalization across multiple aspects. We compare SimHum on Stack Bowls Two trained with the full human dataset (Full) vs. subsets excluding specific factors (w/o Factor). Using the background factor ($\mathcal{F}_{bg}$) as a representative example, Figure \ref{['fig:factor_ablation_example']} illustrates the workflow for filtering human data and designing the corresponding OOD evaluation scenarios.
• Figure 5: Simulation data enhances the generalization to unseen positions. We evaluate position generalization by comparing HumReal and SimHum on a discretized $4\times4$ grid. For each position, we perform 10 evaluation trials and report the average Progress Rate. The black box outlines the region covered by real-robot training data, while the exterior represents uncovered positions.