Memory-Guided View Refinement for Dynamic Human-in-the-loop EQA

TL;DR

DIVRR (Dynamic-Informed View Refinement and Relevance-guided Adaptive Memory Selection), a training-free framework that couples relevance-guided view refinement with selective memory admission that improves robustness under occlusions while preserving fast inference with compact memory is presented.

Abstract

Embodied Question Answering (EQA) has traditionally been evaluated in temporally stable environments where visual evidence can be accumulated reliably. However, in dynamic, human-populated scenes, human activities and occlusions introduce significant perceptual non-stationarity: task-relevant cues are transient and view-dependent, while a store-then-retrieve strategy over-accumulates redundant evidence and increases inference cost. This setting exposes two practical challenges for EQA agents: resolving ambiguity caused by viewpoint-dependent occlusions, and maintaining compact yet up-to-date evidence for efficient inference. To enable systematic study of this setting, we introduce DynHiL-EQA, a human-in-the-loop EQA dataset with two subsets: a Dynamic subset featuring human activities and temporal changes, and a Static subset with temporally stable observations. To address the above challenges, we present DIVRR (Dynamic-Informed View Refinement and Relevance-guided Adaptive Memory Selection), a training-free framework that couples relevance-guided view refinement with selective memory admission. By verifying ambiguous observations before committing them and retaining only informative evidence, DIVRR improves robustness under occlusions while preserving fast inference with compact memory. Extensive experiments on DynHiL-EQA and the established HM-EQA dataset demonstrate that DIVRR consistently improves over existing baselines in both dynamic and static settings while maintaining high inference efficiency.

Figures (6)

• Figure 1: Overview of the evidence verification process in DIVRR. Numerical values denote question-conditioned relevance scores $s_t$. Purple boxes indicate observations admitted into the Relevance-guided Memory via Adaptive Admission Control. At waypoint 7, DIVRR triggers View Refinement and performs multi-view augmentation to select the most relevant viewpoint before memory commitment. The final answer is then generated by conditioning on the compact memory, successfully grounding the response to the question.
• Figure 2: The construction process of the DynHiL-EQA dataset.
• Figure 3: Overview of the DynHiL-EQA dataset statistics.
• Figure 4: The architectural overview of the DIVRR framework, illustrating the coupling of relevance-guided view refinement via multi-view augmentation and selective memory admission.
• Figure 5: Comparison of exploration trajectories. Unlike coverage-centric strategies, DIVRR actively interrogates task-relevant viewpoints to resolve occlusions, leading to verified evidence acquisition.