A Brief Survey on Leveraging Large Scale Vision Models for Enhanced Robot Grasping
Abhi Kamboj, Katherine Driggs-Campbell
TL;DR
This survey examines how large-scale visual pretraining can enhance robot grasping by addressing two core challenges: insufficient object understanding and limited labeled data. It reviews three main visual-pretraining approaches—affordance-map initialization from passive vision tasks, MAE-based pretraining on vast ego-centric hand–object videos with a frozen encoder, and time-contrastive, video-language pretraining on diverse datasets—to improve sample efficiency and transfer to new tasks. The discussion highlights potential future directions, including robust 2D-to-3D affordance representations, 3D grounding, RL-guided training with pretrained visuals, and end-to-end architectures that integrate perception and control. Overall, the paper argues that large-scale visual pretraining can significantly accelerate robust, generalizable robotic grasping in practical settings.
Abstract
Robotic grasping presents a difficult motor task in real-world scenarios, constituting a major hurdle to the deployment of capable robots across various industries. Notably, the scarcity of data makes grasping particularly challenging for learned models. Recent advancements in computer vision have witnessed a growth of successful unsupervised training mechanisms predicated on massive amounts of data sourced from the Internet, and now nearly all prominent models leverage pretrained backbone networks. Against this backdrop, we begin to investigate the potential benefits of large-scale visual pretraining in enhancing robot grasping performance. This preliminary literature review sheds light on critical challenges and delineates prospective directions for future research in visual pretraining for robotic manipulation.