ELBA: Learning by Asking for Embodied Visual Navigation and Task Completion

TL;DR

This work introduces ELBA, an Embodied Learning-By-Asking framework that enables agents to learn when and what to ask to resolve ambiguities during vision-language navigation and task completion. The architecture combines an Actioner, Planner, QA Generator, and QA Evaluator to generate and select informative open-form questions, triggered by uncertainty measured via entropy or gradient magnitude. Evaluation on TEACh shows ELBA improves task and goal-condition success over non-questioning baselines, with notable reductions in the number of questions needed and robust performance across seen and unseen environments. The findings highlight the practical value of interactive QA in embodied AI and outline avenues for more robust, temporally-aware, and memory-enabled QA in future work.

Abstract

The research community has shown increasing interest in designing intelligent embodied agents that can assist humans in accomplishing tasks. Although there have been significant advancements in related vision-language benchmarks, most prior work has focused on building agents that follow instructions rather than endowing agents the ability to ask questions to actively resolve ambiguities arising naturally in embodied environments. To address this gap, we propose an Embodied Learning-By-Asking (ELBA) model that learns when and what questions to ask to dynamically acquire additional information for completing the task. We evaluate ELBA on the TEACh vision-dialog navigation and task completion dataset. Experimental results show that the proposed method achieves improved task performance compared to baseline models without question-answering capabilities.

Figures (10)

• Figure 1: Embodied Learning-By-Asking. An example task "Boil Potato" which involves an agent (left) and an oracle (right). The goal for the agent is to complete the task by navigating and interacting with the environment. When uncertain about the next action, the agent can ask questions to the oracle, receive guidance, and proceed with more confidence to accomplish the task.
• Figure 2: Embodied Learning-By-Asking ( ELBA). At every time step $t$, the Actioner encodes the state information $s_{t-1}$ and outputs the action and object distribution $(p_t^\alpha, p_t^o)$. The confusion module then determines the agent's confusion level by measuring either the entropy of the predicted distribution or the model gradient magnitude. If the confusion level exceeds a certain threshold, the agent will try to ask a question. Based on the state history, the Planner predicts high-level future sub-goal instructions which are later used to generate candidate answers. The QA generator then creates a set of candidate question-answer pairs based on the Planner outputs. The QA evaluator assigns a score to each QA pair, indicating their suitability for the current state, and ranks all QA pairs. The agent samples a pair from the top-$k$ QA pairs and asks the corresponding question if the confusion level decreases after incorporating the chosen QA pair.
• Figure 3: Performance of ELBA -- Oracle QA on question timing. For ELBA w/F model variants, we control the number of fixed time steps the Actioner needs to execute before asking a question. Dashed lines show the performance of ELBA w/E with the proposed confusion module, while solid lines present ELBA w/F model variations with fixed time steps of asking questions.
• Figure 4: Varying confusion thresholds. Performance of (a) entropy-based ( ELBA w/E) and (b) gradient-based ( ELBA w/G) using different thresholds for action and object distributions.
• Figure 5: Average percentage of asked questions for oracle QA and model-generated QA in successful episodes with (a) entropy-based ( ELBA w/E) and (b) gradient-based ( ELBA w/G).