DarkEQA: Benchmarking Vision-Language Models for Embodied Question Answering in Low-Light Indoor Environments

TL;DR

DarkEQA addresses the gap of evaluating embodied question answering under real-world low-light conditions by introducing a physics-based, RAW-space degradation pipeline and a deterministic QA generation process. The benchmark provides 9.4k QA pairs across 52 indoor HM3D-Sem scenes, enabling precise attribution of robustness bottlenecks to illumination and sensor noise. Through extensive evaluation of open and closed VLMs plus an LLIE baseline, the study reveals that current VLMs are brittle to low-light degradation and that LLIE pre-processing yields inconsistent improvements. The work offers a reproducible framework and dataset to drive robustness-focused development for vision-language embodied agents in dark environments.

Abstract

Vision Language Models (VLMs) are increasingly adopted as central reasoning modules for embodied agents. Existing benchmarks evaluate their capabilities under ideal, well-lit conditions, yet robust 24/7 operation demands performance under a wide range of visual degradations, including low-light conditions at night or in dark environments--a core necessity that has been largely overlooked. To address this underexplored challenge, we present DarkEQA, an open-source benchmark for evaluating EQA-relevant perceptual primitives under multi-level low-light conditions. DarkEQA isolates the perception bottleneck by evaluating question answering from egocentric observations under controlled degradations, enabling attributable robustness analysis. A key design feature of DarkEQA is its physical fidelity: visual degradations are modeled in linear RAW space, simulating physics-based illumination drop and sensor noise followed by an ISP-inspired rendering pipeline. We demonstrate the utility of DarkEQA by evaluating a wide range of state-of-the-art VLMs and Low-Light Image Enhancement (LLIE) models. Our analysis systematically reveals VLMs' limitations when operating under these challenging visual conditions. Our code and benchmark dataset will be released upon acceptance.

Figures (7)

• Figure 1: Illustration of the DarkEQA benchmark. Traditional Embodied Question Answering (EQA) primarily evaluates VLMs on well-lit images, overlooking their robustness to real-world low-light conditions. We present DarkEQA, a new benchmark designed to address this evaluation void. DarkEQA assesses VLM performance under two distinct conditions: clean, well-lit inputs (L0) and a multi-level ladder of physics-based low-light images (L1-L5). This heterogeneous design enables a clear analysis of both commonsense reasoning and robustness to visual degradation. Furthermore, the benchmark examines the effect of applying Low-Light Image Enhancement (LLIE) models as a pre-processing step.
• Figure 2: Low-light synthesis pipeline with disentangled illumination and noise factors. To generate controlled low-light inputs for our benchmark, we adopt an ISP-inspired unprocessing and noise formulation from prior work brooks2019unprocessingwei2021physics. Crucially, we produce paired variants for each original image to disentangle failure sources in VLM-based EQA: (a) a physics-based branch (top) that unprocesses sRGB to Bayer RAW, injects four noise components in RAW, and then applies EV drop and gamma compression; and (b) a noise-free branch (bottom) that applies the same EV drop in linear RGB without noise injection. This paired design enables separate evaluation of performance degradation due to illumination reduction versus sensor noise. The bottom-left panel summarizes the sRGB$\rightarrow$RAW unprocessing steps, and the bottom-right panel visualizes the four noise components (shot, read, row-pattern, and quantization noise) as independent signals. The small red boxes in the read and row noise examples indicate zoomed-in crops for visualization.
• Figure 3: Example low-light image synthesization. Synthesized low-light image examples across degradation levels L0–L5. The top row shows EV drop only, while the bottom row shows EV drop combined with noise injection. The lower-right insets show $1/4$-image crops with pixel intensities amplified for visibility; the numbers (×10, ×20, ×50) indicate the amplification factor.
• Figure 4: Question family of our DarkEQA benchmark. Five DarkEQA question categories with examples. DarkEQA encompasses questions asking room-type recognition, room affordance check, object recognition, object attribute.
• Figure 5: Statistics of our DarkEQA benchmark. Dataset statistics, including semantic-class coverage, room-category distribution, and question-category distribution.