Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Acoustic Field Video for Multimodal Scene Understanding

Daehwa Kim, Chris Harrison

TL;DR

Acoustic field video introduces a spatially grounded acoustic modality that visualizes where sound occurs in a scene and aligns with RGB video to enrich vision–language reasoning. The authors implement a real-time beamforming pipeline using commodity MEMS microphone arrays and evaluate its impact on zero-shot scene understanding with a 402 QA benchmark, showing a notable performance gain from $38.3\%$ to $67.4\%$ accuracy when AFV is combined with conventional inputs. They open-source both the dataset and pipeline, demonstrating practical applicability to wearables, robotics, and smart environments while discussing limitations and avenues for future improvements, such as learned spatial encoders and broader environmental testing. Overall, acoustic field video offers a practical and effective path toward more perceptually grounded multimodal intelligence that leverages widely available sensing hardware.

Abstract

We introduce and explore a new multimodal input representation for vision-language models: acoustic field video. Unlike conventional video (RGB with stereo/mono audio), our video stream provides a spatially grounded visualization of sound intensity across a scene, offering a new and powerful dimension of perceptual understanding. Our real-time pipeline uses low-cost beamforming microphone arrays that are already common in smart speakers and increasingly present in robotics and XR headsets, yet this sensing capability remains unutilized for scene understanding. To assess the value of spatial acoustic information, we constructed an evaluation set of 402 question-answer scenes, comparing a state-of-the-art VLM given conventional video with and without paired acoustic field video. Results show a clear and consistent improvement when incorporating spatial acoustic data; the VLM we test improves from 38.3% correct to 67.4%. Our findings highlight that many everyday scene understanding tasks remain underconstrained when relying solely on visual and audio input, and that acoustic field data provides a promising and practical direction for multimodal reasoning. A video demo is available at https://daehwakim.com/seeingsound

Acoustic Field Video for Multimodal Scene Understanding

TL;DR

Acoustic field video introduces a spatially grounded acoustic modality that visualizes where sound occurs in a scene and aligns with RGB video to enrich vision–language reasoning. The authors implement a real-time beamforming pipeline using commodity MEMS microphone arrays and evaluate its impact on zero-shot scene understanding with a 402 QA benchmark, showing a notable performance gain from to accuracy when AFV is combined with conventional inputs. They open-source both the dataset and pipeline, demonstrating practical applicability to wearables, robotics, and smart environments while discussing limitations and avenues for future improvements, such as learned spatial encoders and broader environmental testing. Overall, acoustic field video offers a practical and effective path toward more perceptually grounded multimodal intelligence that leverages widely available sensing hardware.

Abstract

We introduce and explore a new multimodal input representation for vision-language models: acoustic field video. Unlike conventional video (RGB with stereo/mono audio), our video stream provides a spatially grounded visualization of sound intensity across a scene, offering a new and powerful dimension of perceptual understanding. Our real-time pipeline uses low-cost beamforming microphone arrays that are already common in smart speakers and increasingly present in robotics and XR headsets, yet this sensing capability remains unutilized for scene understanding. To assess the value of spatial acoustic information, we constructed an evaluation set of 402 question-answer scenes, comparing a state-of-the-art VLM given conventional video with and without paired acoustic field video. Results show a clear and consistent improvement when incorporating spatial acoustic data; the VLM we test improves from 38.3% correct to 67.4%. Our findings highlight that many everyday scene understanding tasks remain underconstrained when relying solely on visual and audio input, and that acoustic field data provides a promising and practical direction for multimodal reasoning. A video demo is available at https://daehwakim.com/seeingsound
Paper Structure (27 sections, 4 figures)

This paper contains 27 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal and Vision-Language Models
  4. Modalities Beyond RGB Video & Audio
  5. Audio Scene Understanding
  6. Implementation
  7. Hardware
  8. Conventional RGB Video and Audio Streams
  9. Acoustic Field Video Software Pipeline
  10. Zero-shot Scene Understanding
  11. Live Mode
  12. Latency
  13. Potential Applications
  14. Worn Devices (Ego-Centric View)
  15. Mobile Robots
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Many everyday scene-understanding tasks remain underconstrained when relying solely on conventional video (with audio) input. We show that adding acoustic field video, a modality that visualizes the spatial distribution of sound, significantly improves multimodal reasoning. Moreover, the necessary hardware can be practically integrated into many platforms, from smart glasses to robots.
  • Figure 2: Top: Example scenes (conventional RGB and acoustic field still frames shown) drawn from our test set. Bottom table: Example prompts, along with output from Gemini given either Conventional Video or Conventional + Acoustic Field Video as input.
  • Figure 3: QA scene understanding with and without acoustic field video. Left: Overall answer accuracy when the VLM sees only Conventional Video (CV) versus Conventional + Acoustic Field Video (CV+AFV); Middle: Breakdown of correctness. Right: Human raters’ answer preferences.
  • Figure 4: Examples of good attention and failure cases (separated by black horizontal rule). Top: Example scenes (Conventional + Acoustic Field Video) drawn from our test set. Bottom table: Example prompts, along with output from Gemini given either Conventional Video or Conventional + Acoustic Field Video as input.