PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments

TL;DR

This work addresses the limitation of static visual reasoning in partially observable environments by introducing Active Visual Reasoning (AVR), a closed-loop perception–reasoning–action paradigm. It formalizes AVR as a higher-order MDP, builds the CLEVR-AVR benchmark and AVR-152k dataset with explicit Chain-of-Thought supervision, and proposes PhysVLM-AVR, a multimodal LLM-based model trained through a staged strategy to perform active information gathering and reasoning. Key findings show that while current embodied MLLMs can detect information incompleteness, they struggle to actively acquire and integrate new information, and that AVR-Core with CoT supervision is crucial for high performance on AVR tasks. The work advances embodied and interactive visual reasoning, offering benchmarks and datasets that help bridge static reasoning and embodied intelligence, with future work aiming to improve action selection and information-gain prediction in more complex real-world scenarios.

Abstract

Visual reasoning in multimodal large language models (MLLMs) has primarily been studied in static, fully observable settings, limiting their effectiveness in real-world environments where information is often incomplete due to occlusion or limited field of view. Humans, in contrast, actively explore and interact with their environment-moving, examining, and manipulating objects-to gather information through a closed-loop process integrating perception, reasoning, and action. Inspired by this human capability, we introduce the Active Visual Reasoning (AVR) task, extending visual reasoning to partially observable, interactive environments. AVR necessitates agents to: (1) actively acquire information via sequential physical actions, (2) integrate observations across multiple steps for coherent reasoning, and (3) dynamically adjust decisions based on evolving visual feedback. To rigorously evaluate AVR, we introduce CLEVR-AVR, a simulation benchmark featuring multi-round interactive environments designed to assess both reasoning correctness and information-gathering efficiency. We present AVR-152k, a large-scale dataset that offers rich Chain-of-Thought (CoT) annotations detailing iterative reasoning for uncertainty identification, action-conditioned information gain prediction, and information-maximizing action selection, crucial for training agents in a higher-order Markov Decision Process. Building on this, we develop PhysVLM-AVR, an MLLM achieving state-of-the-art performance on CLEVR-AVR, embodied reasoning (OpenEQA, RoboVQA), and passive visual reasoning (GeoMath, Geometry30K). Our analysis also reveals that current embodied MLLMs, despite detecting information incompleteness, struggle to actively acquire and integrate new information through interaction, highlighting a fundamental gap in active reasoning capabilities.

Figures (18)

• Figure 1: Passive Visual Reasoning (top) fails with partial views. Active Visual Reasoning (AVR, bottom) actively interacts to gather information for a correct answer.
• Figure 2: Top: CLEVR-AVR Simulator Benchmark, showing distributions of question types, action space, scenes, and examples. Bottom: Higher-order Markov Decision Process (MDP) paradigm for Active Visual Reasoning (AVR).
• Figure 3: AVR-152k Dataset Construction. Left: Workflows for AVR-Caption and AVR-Embodied Reasoning (perception, temporal reasoning). Right: AVR-Core details including its sequential annotation, CoT supervision, and quality verification.
• Figure 4: Results for Embodied and Visual Reasoning Tasks.
• Figure A-1: Examples of the three scene types in the CLEVR-AVR benchmark.