NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queries

TL;DR

This paper presents NoisyEQA, a benchmark and framework for evaluating Embodied Question Answering under noisy, human-provided questions. It defines four noise types—Latent Hallucination, Memory, Perception, and Semantic Noise—and generates 500 noisy questions via an LLM-based pipeline, enabling controlled study of noise effects. To improve robustness, the authors propose a Self-Correction mechanism with two formats (NAP and NACoT) that detect and rectify noise before answering, and they introduce a 1–5 evaluation scale plus Detection Rate and Correction Rate metrics, augmented by LLM-Match scoring. Experiments reveal that current EQA systems struggle with noisy inputs, but Self-Correction substantially enhances performance, though humans still outperform AI on Active Noise, highlighting avenues for future work and practical impact in real-world, noise-prone settings.

Abstract

The rapid advancement of Vision-Language Models (VLMs) has significantly advanced the development of Embodied Question Answering (EQA), enhancing agents' abilities in language understanding and reasoning within complex and realistic scenarios. However, EQA in real-world scenarios remains challenging, as human-posed questions often contain noise that can interfere with an agent's exploration and response, bringing challenges especially for language beginners and non-expert users. To address this, we introduce a NoisyEQA benchmark designed to evaluate an agent's ability to recognize and correct noisy questions. This benchmark introduces four common types of noise found in real-world applications: Latent Hallucination Noise, Memory Noise, Perception Noise, and Semantic Noise generated through an automated dataset creation framework. Additionally, we also propose a 'Self-Correction' prompting mechanism and a new evaluation metric to enhance and measure both noise detection capability and answer quality. Our comprehensive evaluation reveals that current EQA agents often struggle to detect noise in questions, leading to responses that frequently contain erroneous information. Through our Self-Correct Prompting mechanism, we can effectively improve the accuracy of agent answers.

Figures (11)

• Figure 1: (a) A dining table locates in the dining room, a drunk person mistakenly recalls its location, causing the Agent incorrectly makes a fabricated answer. (b) A systematic taxonomy for human noisy questions. (c) Comparison between Agent’s exploration paths under the accurate question(Green) and noisy question(Pink), the latter may be trapped in a loop, causing physical damage.
• Figure 2: The accuracy of Explore-EQA decreases significantly on noisy questions compared to clean questions.
• Figure 3: Noise types in NoisyEQA benchmark: Latent Hallucination Noise, Memory Noise, Perception Noise, and Semantic Noise. Memory Noise is further subdivided into Memory Position Noise, Memory Shape Noise, Memory Color Noise, Memory Counting Noise, and Memory Material Noise.
• Figure 4: (a) Distribution of various types of noisy questions. (b) Distribution of the attributes within Memory Noise.
• Figure 5: (a) original image and description and (b) disturbed image and description.