Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds

TL;DR

This work tackles the limited 3D spatial understanding of Vision-Language-Action models by introducing Any3D-VLA, a plug-in pipeline that unifies simulator, sensor, and model-estimated point clouds to learn domain-agnostic 3D representations and fuse them with 2D features. It presents a detailed 3D-vision module that builds compressed point clouds from RGBD inputs, aligns them to 2D patches, and performs gated 2D–3D fusion within a VLA backbone, trained with a hybrid point-cloud strategy across multiple depth sources. Through extensive simulation and real-world experiments, Any3D-VLA demonstrates superior zero-shot and post-training performance, achieving up to 62.5% average success in real-world zero-shot tasks and 93.3% in post-training tasks, while also improving LIBERO and CALVIN benchmark results. The key finding is that native 3D representations, when fused with 2D semantics and trained with diverse depth inputs, provide more reliable spatial reasoning for manipulation, enabling robust sim-to-real generalization without requiring expensive depth hardware at deployment.

Abstract

Existing Vision-Language-Action (VLA) models typically take 2D images as visual input, which limits their spatial understanding in complex scenes. How can we incorporate 3D information to enhance VLA capabilities? We conduct a pilot study across different observation spaces and visual representations. The results show that explicitly lifting visual input into point clouds yields representations that better complement their corresponding 2D representations. To address the challenges of (1) scarce 3D data and (2) the domain gap induced by cross-environment differences and depth-scale biases, we propose Any3D-VLA. It unifies the simulator, sensor, and model-estimated point clouds within a training pipeline, constructs diverse inputs, and learns domain-agnostic 3D representations that are fused with the corresponding 2D representations. Simulation and real-world experiments demonstrate Any3D-VLA's advantages in improving performance and mitigating the domain gap. Our project homepage is available at https://xianzhefan.github.io/Any3D-VLA.github.io.

Figures (7)

• Figure 1: Zero-shot comparisons in the real world. For the training dataset, Setting 1 utilizes only the simulator point cloud, whereas Setting 2 incorporates both the simulator and multiple model-estimated point clouds (§\ref{['sec:hybrid_depth']}). During inference, RealSense refers to the sensor-based point cloud, while DA3 refers to the point cloud derived from Depth Anything 3 depth predictions. For instance, (Setting 1, RealSense) denotes training on pure simulator point clouds and inferring with RealSense sensor point clouds.
• Figure 2: Example of Task 1: "Move pink tulip to vase".
• Figure 3: Example of Task 2: "Move condiment cup into right slot of cup carrier".
• Figure 4: Large-scale synthetic pre-training RGBD dataset. (a) shows the RGB images rendered by the simulator and the corresponding ground-truth metric depth maps, which are converted into point clouds using fixed camera intrinsics. (b)(c)(d) respectively present the metric depth maps estimated from a single RGB frame by Depth Anything 3, MapAnything, and UniDepthV2, along with the point clouds computed under the same camera intrinsics. To eliminate interference from background textures, we crop the depth maps in the dataset; therefore, the model inputs are RGB images and the cropped point clouds.
• Figure 5: Real-world post-training RGBD dataset example: “Move condiment cup into right slot of cup carrier.” (a) The RGB image captured by the RealSense camera and its corresponding metric depth map, with the depth converted into a point cloud using fixed camera intrinsics. (b)(c)(d) show the metric depth maps estimated from a single RGB frame by Depth Anything 3, MapAnything, and UniDepthV2, respectively, along with the corresponding point clouds computed using the same camera intrinsics. To eliminate background interference, we crop the depth maps in the dataset; therefore, the model input consists of the RGB image and the cropped point cloud. As can be seen, compared with the model-estimated point clouds, the point clouds reconstructed by RealSense are coarser, which is particularly evident in regions containing transparent objects.