AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models

TL;DR

The paper addresses fine-grained 3D embodied reasoning by grounding referred affordance elements and predicting structured motion triplets conditioned on natural language instructions. It introduces AffordBot, which fuses 3D perception with Multimodal Large Language Models using a holistic surround-view representation and a task-specific chain-of-thought reasoning pipeline (active view selection, grounding, and motion inference) to output triplets $\{(M_i,t_i,a_i)\}_{i=1}^N$. On SceneFun3D, AffordBot achieves state-of-the-art results for both grounding and motion estimation, with ablations validating the contributions of enriched visuals, adaptive labeling, and active perception. Overall, the approach bridges 3D perception and language-driven reasoning to enable instruction-conditioned, physically plausible manipulation in real 3D environments, advancing human-agent collaboration in complex spaces.

Abstract

Effective human-agent collaboration in physical environments requires understanding not only what to act upon, but also where the actionable elements are and how to interact with them. Existing approaches often operate at the object level or disjointedly handle fine-grained affordance reasoning, lacking coherent, instruction-driven grounding and reasoning. In this work, we introduce a new task: Fine-grained 3D Embodied Reasoning, which requires an agent to predict, for each referenced affordance element in a 3D scene, a structured triplet comprising its spatial location, motion type, and motion axis, based on a task instruction. To solve this task, we propose AffordBot, a novel framework that integrates Multimodal Large Language Models (MLLMs) with a tailored chain-of-thought (CoT) reasoning paradigm. To bridge the gap between 3D input and 2D-compatible MLLMs, we render surround-view images of the scene and project 3D element candidates into these views, forming a rich visual representation aligned with the scene geometry. Our CoT pipeline begins with an active perception stage, prompting the MLLM to select the most informative viewpoint based on the instruction, before proceeding with step-by-step reasoning to localize affordance elements and infer plausible interaction motions. Evaluated on the SceneFun3D dataset, AffordBot achieves state-of-the-art performance, demonstrating strong generalization and physically grounded reasoning with only 3D point cloud input and MLLMs.

Figures (10)

• Figure 1: We propose fine-grained 3D embodied reasoning: given a 3D scene and a language task instruction, the agent must identify relevant affordance elements and predict a structured triplet for each: its 3D mask, motion type, and motion axis direction.
• Figure 2: AffordBot Overview. Our method first constructs a holistic multimodal representation designed to bridge 3D scenes with 2D-native MLLMs. This process involves view synthesis, extraction of geometric-semantic descriptors, and their association. Then, our designed Chain-of-Thought (CoT) paradigm guides the MLLM to ultimately predict a structured triplet for the task.
• Figure 3: Illustrations of video-based method limitations: (a) Instructions like "Unplug the Christmas tree lights" or "Adjust the room's temperature using the radiator dial next to the curtain" require anchors (e.g., Christmas tree, curtain) that are missing from the limited video frame. (b) Target objects or parts (e.g., cabinet, door handle) in instructions like "Open the bottom drawer of the wooden cabinet..." or "Open the left part of the window door" are partially visible within the frame.
• Figure 4: AffordBot's Chain-of-Thought Pipeline for Embodied Reasoning. This structured observe-then-infer process leverages multimodal inputs to perform: (1) Active View Selection to identify the most informative view, which may involve zooming in to better see the details of the images, followed by (2) Affordance Grounding to localize target elements, and finally (3) Motion Estimation to infer the required action details.
• Figure 5: Qualitative Results. The figure showcases visual examples of AffordBot performing fine-grained grounding. The illustrated examples include: (1) "Turn on the TV using the remote control on the table." (2) "Open the middle drawer of the TV stand." (3) "Close the bedroom door." (4) "Open the window above the radiator". Please zoom in digitally to view more details.