Avi: Action from Volumetric Inference

TL;DR

Avi reframes robotic action generation as a 3D volumetric inference problem, leveraging a 3D multi-modal language model to predict delta point clouds conditioned on natural language and scene geometry. Actions are derived not by token prediction but through classical geometric optimization, specifically ICP, to align predicted and target volumes and obtain executable trajectories via inverse kinematics. The approach introduces location quantization to extend 3D MLLMs like ShapeLLM-Omni to object-level grounding, improving spatial robustness under occlusion and viewpoint changes. Preliminary experiments on drawer-closing tasks demonstrate semantically consistent, physically realizable action rollouts and highlight the potential of 3D vision-language reasoning as a scalable foundation for robust robotic systems.

Abstract

We propose Avi, a novel 3D Vision-Language-Action (VLA) architecture that reframes robotic action generation as a problem of 3D perception and spatial reasoning, rather than low-level policy learning. While existing VLA models primarily operate on 2D visual inputs and are trained end-to-end on task-specific action policies, Avi leverages 3D point clouds and language-grounded scene understanding to compute actions through classical geometric transformations. Most notably, Avi does not train on previous action tokens, rather, we build upon a 3D Multi-modal Large Language Model (MLLM) to generate the next point cloud and explicitly calculate the actions through classical transformations. This approach enables generalizable behaviors that are robust to occlusions, camera pose variations, and changes in viewpoint. By treating the robotic decision-making process as a structured reasoning task over 3D representations, Avi bridges the gap between high-level language instructions and low-level actuation without requiring opaque policy learning. Our preliminary results highlight the potential of 3D vision-language reasoning as a foundation for scalable, robust robotic systems. Check it out at https://avi-3drobot.github.io/.

Figures (6)

• Figure 1: The left image represents the starting position of the scene. The green voxels represent the predicted next time stamp. The right image represents the end time stamp, and the series of images in between indicate the rollout.
• Figure 2: Comparison against related work. (1) describes Shape LLM Omni. (2) describes our work, Avi. (3) describes the Unified Video Action Model. (4) describes Robot 4D Generation. (5) describes 3D Foundation Policy, FP3.
• Figure 3: Overview of our Vision-Language Action Model Avi, a volumetric vision-language system for robotic action generation. Avi combines stereo reconstruction, 2D segmentation (via Segment Anything), and a fine-tuned 3D Vision-Language Model (based on Qwen-VL and 3D VQVAE embeddings) to predict goal-conditioned 3D volumes. We further align these volumes using classical geometric optimization (ICP) to produce interpretable, spatially grounded actions.
• Figure 4: The 3D MLLM training stage with the new extended vocabulary is detailed, along with the discrete tokens represented between t and t+1. The new Location Quanization tokens are detailed, including the Transformation and Scale Tokens.
• Figure 5: The demonstration expert rollouts provided by LIBERO Goal.