ViRAC: A Vision-Reasoning Agent Head Movement Control Framework in Arbitrary Virtual Environments

TL;DR

ViRAC combines Vision-Language Models for perception with Large-Language Models for reasoning to generate natural, context-aware agent head rotations in arbitrary virtual environments. It introduces a Perception module with a Foundational Memory, and a Decision-making module with an Action History that guides sub-task decomposition via prompts informed by human data. Experiment 1 collects human head-rotation data to ground prompts, while objective and subjective evaluations show ViRAC more closely matches human trajectories and perceived realism than a strong baseline. The framework demonstrates robust generalization across diverse scenes and highlights the value of cognitive reasoning in believable virtual agent behavior, with avenues for multimodal and navigational integration in future work.

Abstract

Creating lifelike virtual agents capable of interacting with their environments is a longstanding goal in computer graphics. This paper addresses the challenge of generating natural head rotations, a critical aspect of believable agent behavior for visual information gathering and dynamic responses to environmental cues. Although earlier methods have made significant strides, many rely on data-driven or saliency-based approaches, which often underperform in diverse settings and fail to capture deeper cognitive factors such as risk assessment, information seeking, and contextual prioritization. Consequently, generated behaviors can appear rigid or overlook critical scene elements, thereby diminishing the sense of realism. In this paper, we propose \textbf{ViRAC}, a \textbf{Vi}sion-\textbf{R}easoning \textbf{A}gent Head Movement \textbf{C}ontrol framework, which exploits the common-sense knowledge and reasoning capabilities of large-scale models, including Vision-Language Models (VLMs) and Large-Language Models (LLMs). Rather than explicitly modeling every cognitive mechanism, ViRAC leverages the biases and patterns internalized by these models from extensive training, thus emulating human-like perceptual processes without hand-tuned heuristics. Experimental results in multiple scenarios reveal that ViRAC produces more natural and context-aware head rotations than recent state-of-the-art techniques. Quantitative evaluations show a closer alignment with real human head-movement data, while user studies confirm improved realism and cognitive plausibility.

Figures (14)

• Figure 1: Visualization example of our framework's head-turn in a busy shopping mall scenario. The top row shows the agent’s third-person view, while the middle row depicts the corresponding first-person view. Each selected action (e.g., "Looking at Mall Map”) is annotated with the rationale ("To understand the mall layout and key locations”), showing how ViRAC’s cognitive reasoning and visual perception modules interact to produce context-aware head rotations that closely resemble human behavior.
• Figure 2: Categorized distribution of participants’ self-reported head-movement rationales.
• Figure 3: ViRAC Framework. This is an example of 'Street'. VLM identifies salient objects from the scene and updates the Foundational Memory Module (FMM). LLM then references both the Action History Module (AHM) and the FMM to decompose high-level cognitive goals into sub-tasks, guiding context-sensitive actions.
• Figure 4: Mean scores and standard deviations for each metric, rated on a seven-point Likert scale. Higher values denote more favorable judgments. The brackets indicate statistically significant differences (**:$p$ < 0.01). ViRAC achieves results comparable to Human in most metrics and consistently outperforms Track.
• Figure 5: Sample frames from the crosswalk and mall scenarios.