Learning to Grasp Anything by Playing with Random Toys

TL;DR

This work tackles the challenge of generalizing robotic grasping to unseen real-world objects. It introduces LEGO, a framework that trains on random configurations of four shape primitives (spheres, cuboids, cylinders, rings) and uses an object-centric DetPool representation to enable zero-shot generalization to real objects. A Cézanne-inspired toy dataset (250 toys, 1–5 primitives per toy, 2,500 simulated and 2,000 real demonstrations across platforms) supports rigorous evaluation, with LEGO achieving a $67\%$ zero-shot success rate on 64 YCB objects in simulation and real-world tests, outperforming larger, pre-trained baselines. The findings reveal scaling laws where demonstration count drives generalization and demonstrate cross-embodiment transfer to dexterous hands, offering a scalable path toward robust, generalizable robotic manipulation.

Abstract

Robotic manipulation policies often struggle to generalize to novel objects, limiting their real-world utility. In contrast, cognitive science suggests that children develop generalizable dexterous manipulation skills by mastering a small set of simple toys and then applying that knowledge to more complex items. Inspired by this, we study if similar generalization capabilities can also be achieved by robots. Our results indicate robots can learn generalizable grasping using randomly assembled objects that are composed from just four shape primitives: spheres, cuboids, cylinders, and rings. We show that training on these "toys" enables robust generalization to real-world objects, yielding strong zero-shot performance. Crucially, we find the key to this generalization is an object-centric visual representation induced by our proposed detection pooling mechanism. Evaluated in both simulation and on physical robots, our model achieves a 67% real-world grasping success rate on the YCB dataset, outperforming state-of-the-art approaches that rely on substantially more in-domain data. We further study how zero-shot generalization performance scales by varying the number and diversity of training toys and the demonstrations per toy. We believe this work offers a promising path to scalable and generalizable learning in robotic manipulation. Demonstration videos, code, checkpoints and our dataset are available on our project page: https://lego-grasp.github.io/ .

Figures (7)

• Figure 1: Our grasping policy built from just four basic primitives (left), trained exclusively on random toy compositions (middle), zero-shot generalizes to real-world objects (right) and achieves an 67% success rate on 64 objects from the YCB dataset.
• Figure 2: Our Cézanne toys are composed of different number of primitives. We generate each toy by randomly assembling 1-5 primitives and randomizing dimensions and colors.
• Figure 3: The LEGO architecture with DetPool.(a) LEGO uses a ViT with DetPool to extract features of the target object and uses a transformer to predict future actions based on the visual features and the proprioception. (b) The ViT extracts features that focus on the target object via DetPool which restrains the attention to the object patches using an attention mask and performs mean pooling on the output object patch tokens to get the final object-centric vision feature.
• Figure 4: Scaling studies.Left: The zero-shot success rate scales with both the number of demos and the number of unique toys. We also find that once the number of demos is sufficient, 25 toys is already enough to achieve a robust zero-shot transfer. Right: The performances scales with the size of the policy transformer until it saturates at the size of 86M.
• Figure 5: Hardware Configuration for Franka Emika Panda with Robotiq Gripper.