Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning

TL;DR

The paper tackles the problem of ambiguous human instructions for embodied home robots by introducing the Ask-to-Act task and proposing AutoAsk, a method that fine-tunes a multimodal large language model into a vision-language-action policy trained with online reinforcement learning using LLM-generated rewards. This approach eliminates the need for large human demonstrations or hand-designed rewards and demonstrates clear performance gains over strong baselines, including zero-shot GPT-4o and SFT-based MLLMs, on unseen scenes and tasks. A key contribution is using per-step LLM rewards to teach the agent to both act and ask relevant clarification questions, enabling robust reasoning under partial observability. The work shows that dense, context-aware LLM rewards can effectively train embodied agents to resolve ambiguity through dialogue and interaction, with promising generalization to novel environments and task configurations.

Abstract

Embodied agents operating in household environments must interpret ambiguous and under-specified human instructions. A capable household robot should recognize ambiguity and ask relevant clarification questions to infer the user intent accurately, leading to more effective task execution. To study this problem, we introduce the Ask-to-Act task, where an embodied agent is tasked with a single or multi-object rearrangement task using an under-specified instruction in a home environment. The agent must strategically ask minimal, yet relevant, clarification questions to resolve ambiguity while navigating under partial observability. To address this challenge, we propose a novel approach that fine-tunes multi-modal large language models (MLLMs) as vision-language-action (VLA) policies using online reinforcement learning (RL) with LLM-generated rewards. Our method eliminates the need for large-scale human demonstrations or manually engineered rewards for training such agents. We benchmark against strong zero-shot baselines including GPT-4o as well as supervised fine-tuned MLLMs on our task. Our results show that our RL-finetuned MLLM outperforms all baselines by a significant margin (10.4-16.5%), generalizing well to novel scenes and tasks. To the best of our knowledge, this is the first demonstration of adapting MLLMs as VLA agents that can act and ask for help using LLM-generated rewards with online RL.

Figures (7)

• Figure 1: Ask-to-Act task. In this task, the user requests a specific green cup but, instead of describing it in detail, asks the agent, "Bring the cup and place it on coffee table". Since the user's intent is unclear, an agent must ask a minimum number of clarification questions to disambiguate the requested object (e.g."Are you looking for a red cup?" or "Is it on the kitchen counter?"). We consider under-specified single and multi-object rearrangement tasks that involve inquiring about user preferences and resolving different types of ambiguities, about object attributes, spatial relationships, object size, placement location, or combinations of the four.
• Figure 2: MLLM Policy Architecture. The policy takes as input a task instruction, past observations, actions, user response to questions asked and outputs a high-level action or a question in natural language.
• Figure 3: Analysis. (a.) Task Performance vs. Budget of Questions. Evaluation performance of policies trained under different budget of questions vs. task Success Rate and Ambiguity Resolution Efficiency score. (c.) Success Rate vs. Number of Required and Asked Questions. Evaluation performance of SFT and RL trained policies vs. number of required and total questions asked by the agent.
• Figure 4: Qualitative Example. Successful trajectory on an evaluation episode from Unseen Tasks split.
• Figure 5: Qualitative Example. Successful trajectories of our method on $2$ evaluation episodes from Unseen Tasks split.