Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation

TL;DR

ToolEQA introduces external tool augmentation with multi-step reasoning for Embodied Question Answering in 3D environments. The architecture couples a planner, controller, and executor to decompose tasks, reason about tool usage, and gather observations through executable tools, enabling shorter, more accurate exploration. A scalable EQA data generation pipeline produces EQA-RT, an 18K-question dataset with validated reasoning trajectories to train and evaluate the agent. Across EQA-RT-Seen/Unseen and established benchmarks like HM-EQA, OpenEQA, and EXPRESS-Bench, ToolEQA achieves significant gains in success rate and efficiency, demonstrating the value of explicit reasoning and tool usage in embodied AI tasks.

Abstract

Embodied Question Answering (EQA) requires agents to explore 3D environments to obtain observations and answer questions related to the scene. Existing methods leverage VLMs to directly explore the environment and answer questions without explicit thinking or planning, which limits their reasoning ability and results in excessive or inefficient exploration as well as ineffective responses. In this paper, we introduce ToolEQA, an agent that integrates external tools with multi-step reasoning, where external tools can provide more useful information for completing the task, helping the model derive better exploration directions in the next step of reasoning and thus obtaining additional effective information. This enables ToolEQA to generate more accurate responses with a shorter exploration distance. To enhance the model's ability for tool-usage and multi-step reasoning, we further design a novel EQA data generation pipeline that automatically constructs large-scale EQA tasks with reasoning trajectories and corresponding answers. Based on the pipeline, we collect the EQA-RT dataset that contains about 18K tasks, divided into a training set EQA-RT-Train, and two test sets EQA-RT-Seen (scenes overlapping with the training set) and EQA-RT-Unseen (novel scenes). Experiments on EQA-RT-Seen and EQA-RT-Unseen show that ToolEQA improves the success rate by 9.2~20.2% over state-of-the-art baselines, while outperforming the zero-shot ToolEQA by 10% in success rate. In addition, ToolEQA also achieves state-of-the-art performance on the HM-EQA, OpenEQA, and EXPRESS-Bench datasets, demonstrating its generality. Our homepage see https://tooleqa.github.io.

Figures (7)

• Figure 1: Overview of the proposed ToolEQA for Embodied Question Answering (EQA). ToolEQA enables to decompose questions into structured plans, reasoning to select tools, and invoke tools to explore and answer. ToolEQA achieves highest accuracy with fewer reasoning steps.
• Figure 2: Overview of the ToolEQA agent workflow.
• Figure 3: EQA Data Generation Pipeline.
• Figure 4: Data statistics of the training set (EQA-RT-Train) and two test sets (EQA-RT-Seen and EQA-RT-Unseen). The scenes in EQA-RT-Seen have the overlap with EQA-RT-Train, while the scenes in EQA-RT-Unseen are not present in the training set.
• Figure 5: Illustration of how explicit reasoning guides efficient exploration, enabling ToolEQA to answer questions faster and more accurately.