ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning

TL;DR

This work tackles ambiguous natural-language instructions in embodied search by treating navigation, dialogue, and memory retrieval as a unified, cost-aware decision problem. It introduces ESearch-R1, a framework that uses a Group Relative Policy Optimization (HC-GRPO) to train a multimodal LLM-based agent without a value critic, prioritizing low-cost information gathering before movement. Through ESearch-Bench in AI2-THOR, the approach demonstrates substantial reductions in total task cost while maintaining high success rates, illustrating emergent, cost-aware strategies such as prioritizing memory retrieval and targeted questioning. The results show robust performance across varying ambiguity and cost configurations, with ablations confirming the essential roles of dialogue and memory and the benefits of online optimization. Overall, ESearch-R1 advances cost-efficient, interactive embodied AI by aligning deep reasoning with physical-world constraints and providing a scalable benchmark for future work.

Abstract

Multimodal Large Language Models (MLLMs) have empowered embodied agents with remarkable capabilities in planning and reasoning. However, when facing ambiguous natural language instructions (e.g., "fetch the tool" in a cluttered room), current agents often fail to balance the high cost of physical exploration against the cognitive cost of human interaction. They typically treat disambiguation as a passive perception problem, lacking the strategic reasoning to minimize total task execution costs. To bridge this gap, we propose ESearch-R1, a cost-aware embodied reasoning framework that unifies interactive dialogue (Ask), episodic memory retrieval (GetMemory), and physical navigation (Navigate) into a single decision process. We introduce HC-GRPO (Heterogeneous Cost-Aware Group Relative Policy Optimization). Unlike traditional PPO which relies on a separate value critic, HC-GRPO optimizes the MLLM by sampling groups of reasoning trajectories and reinforcing those that achieve the optimal trade-off between information gain and heterogeneous costs (e.g., navigate time, and human attention). Extensive experiments in AI2-THOR demonstrate that ESearch-R1 significantly outperforms standard ReAct-based agents. It improves task success rates while reducing total operational costs by approximately 50\%, validating the effectiveness of GRPO in aligning MLLM agents with physical world constraints.

Figures (5)

• Figure 1: From passive search to proactive interactive disambiguation. In scenes with multiple candidate instances, traditional agents rely on exhaustive navigation and inspection, incurring high movement cost. ESearch-R1 behaves as a collaborative partner: it first narrows down the candidate set via asking the user and retrieving episodic memory, and then executes the shortest path to the uniquely identified goal, thereby reducing both navigation cost and the number of user interruptions.
• Figure 2: ESearch-R1 architecture and closed-loop inference. Under multimodal context, the unified decision core assigns comparable information–cost trade-offs to Ask/GetMemory/Navigate; new observations from exploration are written into memory, forming a closed perception–memory–decision loop.
• Figure 3: Cost parameter sensitivity analysis. ESearch-R1 (blue) consistently outperforms baselines across all cost configurations, demonstrating robust cost-aware optimization. The vertical dashed lines indicate our default parameter settings.
• Figure 4: Training dynamics during HC-GRPO’s online optimization. Left: average return (task reward minus total cost) steadily increases. Right: average response token length (including CoT and actions) decreases markedly.
• Figure 5: Comparison of decision distributions between ESearch-SFT and ESearch-R1.