Simulating Human Audiovisual Search Behavior

TL;DR

This work tackles audiovisual search under uncertainty, where locating a target relies on integrating noisy auditory and visual cues while managing embodied costs. Sensonaut treats this as a resource-rational decision problem implemented as a POMDP and solved via reinforcement learning to produce human-like search policies. The key contributions are (a) a unified theory linking cue integration with embodied action costs, (b) a POMDP+RL realization that reproduces human search trajectories and error modes, and (c) a VR audiovisual search dataset for validation. The practical impact lies in enabling predictive simulations to optimize XR interfaces and assistive guidance that reduce search cost and cognitive load.

Abstract

Locating a target based on auditory and visual cues$\unicode{x2013}$such as finding a car in a crowded parking lot or identifying a speaker in a virtual meeting$\unicode{x2013}$requires balancing effort, time, and accuracy under uncertainty. Existing models of audiovisual search often treat perception and action in isolation, overlooking how people adaptively coordinate movement and sensory strategies. We present Sensonaut, a computational model of embodied audiovisual search. The core assumption is that people deploy their body and sensory systems in ways they believe will most efficiently improve their chances of locating a target, trading off time and effort under perceptual constraints. Our model formulates this as a resource-rational decision-making problem under partial observability. We validate the model against newly collected human data, showing that it reproduces both adaptive scaling of search time and effort under task complexity, occlusion, and distraction, and characteristic human errors. Our simulation of human-like resource-rational search informs the design of audiovisual interfaces that minimize search cost and cognitive load.

Figures (9)

• Figure 1: Overview of our computational model of embodied audiovisual search. The agent maintains a belief over possible target locations by integrating prior knowledge with audio and visual likelihoods. This belief is updated step by step and passed to a resource-rational policy, which weighs expected utility against the costs of physical effort and time. The policy selects embodied actions (turns, forward steps, staying, or committing) that shape new observations from the environment. Through this loop, the model simulates how people combine cue integration with action costs to efficiently locate targets under uncertainty.
• Figure 2: Example trajectory and belief update of Sensonaut across timesteps. The top row shows the agent’s position and heading (green solid arrow) and a human participant's position and heading (pink dashed arrow) in the map's grid view, with a hidden target (green-highlighted box). The second and third rows show the evolving auditory and visual likelihoods: audition offers omnidirectional evidence while vision provides high precision but view-dependent and occlusion-limited evidence. The bottom row shows the posterior belief (which serves as the prior for the next timestep), integrating both modalities and prior. Over time, embodied actions such as head turns reshape the likelihoods, reduce ambiguity, and concentrate belief mass on the true target location (circled).
• Figure 3: Data collection setup. Before each trial, participants wearing a Meta Quest 3 saw a text instruction indicating the target car color. During the trial, they physically turned and walked in a VR parking lot to locate the sound-emitting target vehicle. To commit, they pressed the 'A' button on the Quest controller, which cleared all cars, and then recorded their estimated target location using a projectile ray selector.
• Figure 4: Human and Sensonaut performance across initial target angle, number of objects, and number of distractors. Each row shows mean accuracy, search time, head turns, and displacement, with shaded regions indicating 95% confidence intervals. Both human and agent show longer search times, more head turns, and more locomotion when the target starts to the side or behind, or with more distractors.
• Figure 5: Correlation between human and Sensonaut on accuracy, search time, cumulative head rotation, and displacement. Sensonaut reproduces map-level patterns in accuracy, search time, and head turns, with weaker correspondence in displacement, further analyzed in Section \ref{['sec:locomotion-misalignment']}.