3D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks

TL;DR

The paper addresses the limited generalization of Vision-Language-Action models to unseen robotic tasks by enriching scene context and reasoning. It introduces 3D-CAVLA, which extends OpenVLA-OFT with a lightweight depth encoder, chain-of-thought narrative prompts, and task-aware ROI pooling, implemented on a two-camera setup with LoRA-tuned LLaMA-2 7B. Empirical results on the LIBERO benchmark show near state-of-the-art in-distribution performance (≈98%) and a substantial absolute gain (8.8%) on unseen tasks, with ablations underscoring the critical role of depth and the benefits of reasoning for long-horizon tasks. The work demonstrates that integrating 3D context and structured reasoning can significantly enhance zero-shot and few-shot transfer in vision-language robotic manipulation, and it provides open-source resources for further community-driven evaluation.

Abstract

Robotic manipulation in 3D requires learning an $N$ degree-of-freedom joint space trajectory of a robot manipulator. Robots must possess semantic and visual perception abilities to transform real-world mappings of their workspace into the low-level control necessary for object manipulation. Recent work has demonstrated the capabilities of fine-tuning large Vision-Language Models (VLMs) to learn the mapping between RGB images, language instructions, and joint space control. These models typically take as input RGB images of the workspace and language instructions, and are trained on large datasets of teleoperated robot demonstrations. In this work, we explore methods to improve the scene context awareness of a popular recent Vision-Language-Action model by integrating chain-of-thought reasoning, depth perception, and task-oriented region of interest detection. Our experiments in the LIBERO simulation environment show that our proposed model, 3D-CAVLA, improves the success rate across various LIBERO task suites, achieving an average success rate of 98.1$\%$. We also evaluate the zero-shot capabilities of our method, demonstrating that 3D scene awareness leads to robust learning and adaptation for completely unseen tasks. 3D-CAVLA achieves an absolute improvement of 8.8$\%$ on unseen tasks. We will open-source our code and the unseen tasks dataset to promote community-driven research here: https://3d-cavla.github.io

Figures (3)

• Figure 1: Our proposed model, 3D-CAVLA, integrates chain-of-thought style narrative task descriptions, depth embeddings and Region Of Interest (ROI) pooling to improve the scene awareness of vision-language-action modeling. While GPT4 and ROI Detection are frozen components, our depth encoder is a lightweight PointNet pointnet inspired trainable network with spatial invariance transformation, convolution blocks and linear projections to project the embeddings to match the input dimensions of LLaMA 2 7B touvron2023llama
• Figure 2: LLM prompt to decompose task instructions into executable steps that can be generalized across seen and unseen tasks.
• Figure 3: Our framework for task aware region of interest detection using entity recognition, object detection and tracking.