FieldGen: From Teleoperated Pre-Manipulation Trajectories to Field-Guided Data Generation

TL;DR

The paper tackles the bottleneck of acquiring large-scale, diverse, high‑quality real‑world robotic manipulation data by separating manipulation into a scalable pre‑manipulation reach and a precise fine manipulation phase. FieldGen constructs a pre‑manipulation field $\oldsymbol{\mathcal{F}}_{Gen}$ from a small set of demonstrations and uses a cone field for position and a spherical field for orientation to autonomously generate diverse reach trajectories, while FieldGen‑Reward adds reward annotations to diversify training signals. Empirical results on the Agibot G1 show FieldGen yields higher policy success, better generalization, and far reduced operator effort compared to teleoperation, with cycloid trajectories and a carefully chosen $eta$ parameter further improving performance. The approach offers scalable, real‑world data generation with stronger coverage and robustness, suggesting a practical path to large‑scale robotic datasets and improved downstream manipulation policies.

Abstract

Large-scale and diverse datasets are vital for training robust robotic manipulation policies, yet existing data collection methods struggle to balance scale, diversity, and quality. Simulation offers scalability but suffers from sim-to-real gaps, while teleoperation yields high-quality demonstrations with limited diversity and high labor cost. We introduce FieldGen, a field-guided data generation framework that enables scalable, diverse, and high-quality real-world data collection with minimal human supervision. FieldGen decomposes manipulation into two stages: a pre-manipulation phase, allowing trajectory diversity, and a fine manipulation phase requiring expert precision. Human demonstrations capture key contact and pose information, after which an attraction field automatically generates diverse trajectories converging to successful configurations. This decoupled design combines scalable trajectory diversity with precise supervision. Moreover, FieldGen-Reward augments generated data with reward annotations to further enhance policy learning. Experiments demonstrate that policies trained with FieldGen achieve higher success rates and improved stability compared to teleoperation-based baselines, while significantly reducing human effort in long-term real-world data collection. Webpage is available at https://fieldgen.github.io/.

Figures (7)

• Figure 1: (a) Cone Field for Position. (b) Spherical Field for Orientation.
• Figure 2: Equal-Time Data Effectiveness Experiment Setups. Rotate Pick requires the end-effector change orientation to pick objects, while Affordance Pick requires to manipulate specific position.
• Figure 3: (a) Equal-Time Data Effectiveness. (b) Equal-Data Generalization.
• Figure 4: Data Quality and Generalization Experiment Setups
• Figure 5: (a) Ablation on Curve. (b) Ablation on $\beta$ Parameter. (c) $XY$ plane scatter plot of Diversity.