ForeRobo: Unlocking Infinite Simulation Data for 3D Goal-driven Robotic Manipulation

TL;DR

ForeRobo introduces ForeGen and ForeFormer to unlock infinite, high-fidelity simulation data for 3D goal-state-driven robotic manipulation. ForeGen generates diverse tasks, scenes, and goal states via GPT-4 and CPCA-based state transfer, creating the ForeMani-v1 dataset. ForeFormer uses conditional diffusion with transformer-based encoders to predict per-point goal states from scene and task, enabling zero-shot sim-to-real with strong generalization across rigid and articulated objects. The approach achieves notable gains over baselines in simulation and real-world experiments and demonstrates scalable, interpretability-friendly planning-driven manipulation. Limitations include handling deformable objects and full preservation of structural details, guiding future work toward more general and robust manipulation in varied environments.

Abstract

Efficiently leveraging simulation to acquire advanced manipulation skills is both challenging and highly significant. We introduce \textit{ForeRobo}, a generative robotic agent that utilizes generative simulations to autonomously acquire manipulation skills driven by envisioned goal states. Instead of directly learning low-level policies, we advocate integrating generative paradigms with classical control. Our approach equips a robotic agent with a self-guided \textit{propose-generate-learn-actuate} cycle. The agent first proposes the skills to be acquired and constructs the corresponding simulation environments; it then configures objects into appropriate arrangements to generate skill-consistent goal states (\textit{ForeGen}). Subsequently, the virtually infinite data produced by ForeGen are used to train the proposed state generation model (\textit{ForeFormer}), which establishes point-wise correspondences by predicting the 3D goal position of every point in the current state, based on the scene state and task instructions. Finally, classical control algorithms are employed to drive the robot in real-world environments to execute actions based on the envisioned goal states. Compared with end-to-end policy learning methods, ForeFormer offers superior interpretability and execution efficiency. We train and benchmark ForeFormer across a variety of rigid-body and articulated-object manipulation tasks, and observe an average improvement of 56.32\% over the state-of-the-art state generation models, demonstrating strong generality across different manipulation patterns. Moreover, in real-world evaluations involving more than 20 robotic tasks, ForeRobo achieves zero-shot sim-to-real transfer and exhibits remarkable generalization capabilities, attaining an average success rate of 79.28\%.

Figures (10)

• Figure 1: Overview of ForeRobo. ForeRobo primarily consists of a data generation component, ForeGen, and a state prediction model, ForeFormer. The top-right part of the figure demonstrates how ForeFormer, trained exclusively on simulation data, enables the robot arm to accomplish manipulation tasks in real-world environments. The process mainly involves five steps: (1) task-relevant object segmentation; (2) object point cloud acquisition; (3) goal state prediction using ForeFormer; (4) grasp detection; and (5) robot motion planning. Detailed descriptions of these steps are provided in the Appendix. The bottom part of the figure illustrates that ForeFormer, trained entirely on simulation data, can be zero-shot transferred to real-world environments and generalize across diverse objects and manipulation tasks.
• Figure 2: Overview of ForeGen and ForeMani-v1. ForeGen consists of the following stages: a) task proposal, b) scene generation, and c) state generation. ForeMani-v1 encompasses 1,536 objects and 106 tasks, with each task containing an average of 21 scenarios and 394 goal states.
• Figure 3: Pipeline of Cross-instance Proximity Contact Alignment (CPCA). Given the demonstrated goal state in a task's demonstration scene, CPCA can generate the corresponding goal states for all augmented scenes of that task.
• Figure 4: Overview of ForeFormer and robot motion planning. PTv3 refers to PointTransformer-v3, and SAT refers to the Self-Attention Transformer. In the task "Lift the kettle above the cup and tilt it to pour water" depicted in the figure, the kettle is the object to be manipulated, which is embedded into the network via the object point cloud encoder (comprising SAT and two MLPs). The cup, being the task-relevant object, is embedded into the network through the background point cloud encoder (which includes PTv3 and an MLP).
• Figure 5: Simulation tasks. We evaluated the performance of ForeFormer and the baseline methods on ten simulated tasks, including six rigid-object manipulation tasks and four articulated-object manipulation tasks. In the captions below each task illustration, the manipulated object is highlighted in blue, followed by the task description. The red arrows indicate the transformation of objects from their initial states to the goal states that satisfy the task description.