ManiBox: Enhancing Spatial Grasping Generalization via Scalable Simulation Data Generation

TL;DR

ManiBox introduces a bounding-box-guided, state-based teacher-student framework to tackle spatial grasping generalization. By training a privileged teacher in simulation to generate large, diverse trajectory data and distilling a robust, bounding-box–driven student for real-world deployment, the method achieves zero-shot transfer across varied objects, poses, and backgrounds. The work uncovers a power-law relationship between spatial volume and required data, and a Michaelis-Menten–like saturation of grasp success with data, underscoring the value of scalable simulation data for spatial generalization. Real-world experiments validate strong generalization capabilities, highlighting ManiBox’s practical potential for robust manipulation in diverse environments.

Abstract

Learning a precise robotic grasping policy is crucial for embodied agents operating in complex real-world manipulation tasks. Despite significant advancements, most models still struggle with accurate spatial positioning of objects to be grasped. We first show that this spatial generalization challenge stems primarily from the extensive data requirements for adequate spatial understanding. However, collecting such data with real robots is prohibitively expensive, and relying on simulation data often leads to visual generalization gaps upon deployment. To overcome these challenges, we then focus on state-based policy generalization and present \textbf{ManiBox}, a novel bounding-box-guided manipulation method built on a simulation-based teacher-student framework. The teacher policy efficiently generates scalable simulation data using bounding boxes, which are proven to uniquely determine the objects' spatial positions. The student policy then utilizes these low-dimensional spatial states to enable zero-shot transfer to real robots. Through comprehensive evaluations in simulated and real-world environments, ManiBox demonstrates a marked improvement in spatial grasping generalization and adaptability to diverse objects and backgrounds. Further, our empirical study into scaling laws for policy performance indicates that spatial volume generalization scales with data volume in a power law. For a certain level of spatial volume, the success rate of grasping empirically follows Michaelis-Menten kinetics relative to data volume, showing a saturation effect as data increases. Our videos and code are available in https://thkkk.github.io/manibox.

Figures (26)

• Figure 1: Overview. We introduce ManiBox, a bbox-guided manipulation method using a teacher-student framework to enhance spatial generalization in grasping policies. We reveal that spatial volume generalization scales with data volume in power law, with grasping success following Michaelis-Menten kinetics relative to data volume for specified spatial volumes. Extensive real-world tests show ManiBox's robust adaptability to varied spatial positions, objects, and backgrounds.
• Figure 2: ManiBox illustration. (a) Utilizing PPO and domain randomization, we train a sophisticated teacher policy in the simulator that utilizes privileged object information to determine actions. (b) The teacher policy generates scalable trajectory data, utilizing bounding box coordinates instead of traditional high-dimensional visual inputs or privileged information. (c) A state-based student policy, generalizable and capable of zero-shot transfer, is trained on this extensive simulation dataset, greatly improving spatial generalization. (d) Guided by bounding boxes, the student policy precisely executes actions for real robots, achieving improved generalization capabilities.
• Figure 3: The scaling relationship between spatial generalization and data volume in the grasping task, measured under different spatial ranges. Data volume represents the number of trajectories used to train the student policy. The estimated 80% success point is marked, with blue points showing the average success rate across three seeds and error bars indicating standard deviation.
• Figure 4: The relationship between spatial volume and data amounts needed to reach 80% grasping success rate. The fitted curve represents a power function $y=640.32\cdot x^{0.35}$.
• Figure 5: Demonstration of the generalization of our methods across different backgrounds, objects, and positions. The h1, h2, h3, h4 are randomly selected object heights in the real world, which are approximately 57cm, 65cm, 72cm, and 65cm. LB, LF, RF, RB, and C are abbreviations for Left-Back, Left-Front, Right-Front, Right-Back, and Center area respectively.

Theorems & Definitions (3)

• Lemma 1: Details are in Appendix \ref{['app_proof1']}
• Lemma 2: Details and proofs are in Appendix \ref{['app_proof2']}
• proof