PointVLA: Injecting the 3D World into Vision-Language-Action Models

TL;DR

PointVLA addresses the limitation of 2D-only Vision-Language-Action models by inserting 3D point cloud information through a modular injector that minimally perturbs the pretrained action expert. By performing a skip-block analysis, it injects 3D features only into blocks deemed less critical, preserving the core 2D representations while leveraging 3D spatial cues. Across simulated and real-world tasks, PointVLA achieves superior few-shot multi-tasking, demonstrates real-vs-photo discrimination, and exhibits height adaptability and robust performance on long-horizon packing tasks. The approach offers a cost-efficient path to 3D-aware robot agents without full retraining of foundational 2D models.

Abstract

Vision-Language-Action (VLA) models excel at robotic tasks by leveraging large-scale 2D vision-language pretraining, but their reliance on RGB images limits spatial reasoning critical for real-world interaction. Retraining these models with 3D data is computationally prohibitive, while discarding existing 2D datasets wastes valuable resources. To bridge this gap, we propose PointVLA, a framework that enhances pre-trained VLAs with point cloud inputs without requiring retraining. Our method freezes the vanilla action expert and injects 3D features via a lightweight modular block. To identify the most effective way of integrating point cloud representations, we conduct a skip-block analysis to pinpoint less useful blocks in the vanilla action expert, ensuring that 3D features are injected only into these blocks--minimizing disruption to pre-trained representations. Extensive experiments demonstrate that PointVLA outperforms state-of-the-art 2D imitation learning methods, such as OpenVLA, Diffusion Policy and DexVLA, across both simulated and real-world robotic tasks. Specifically, we highlight several key advantages of PointVLA enabled by point cloud integration: (1) Few-shot multi-tasking, where PointVLA successfully performs four different tasks using only 20 demonstrations each; (2) Real-vs-photo discrimination, where PointVLA distinguishes real objects from their images, leveraging 3D world knowledge to improve safety and reliability; (3) Height adaptability, Unlike conventional 2D imitation learning methods, PointVLA enables robots to adapt to objects at varying table height that unseen in train data. Furthermore, PointVLA achieves strong performance in long-horizon tasks, such as picking and packing objects from a moving conveyor belt, showcasing its ability to generalize across complex, dynamic environments.

Figures (7)

• Figure 1: Overview of PointVLA framework.Left: The 2D image observation and instruction are processed by the vision-language model. The vanilla action expert remains frozen, while the new point cloud representation is integrated into the action expert through a modular network. Right: Details of the point cloud injector.
• Figure 2: Skip block analysis for action expert in VLA model.Left: skipping only one block at a time. Right: Skipping multiple consecutive blocks starting from the 11th block.
• Figure 3: Setup for bimanual UR5e. We utilize three cameras: two RealSense D435i mounted on the wrists and one RealSense L515 positioned above. Our model is evaluated on a challenging long-horizon task that involves picking up two laundry detergent bottles from a moving conveyor belt and packing them into a box.
• Figure 4: Setup for bimanual AgileX. We utilize three cameras: two RealSense D435i mounted on the wrists and one RealSense L515 positioned above. Our model is evaluated on four tasks in a few-shot setting.
• Figure 5: Experimental results on few-shot multi-tasking on bimanual AgileX.