Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes

TL;DR

The paper addresses the need for fine-grained surgical scene understanding by integrating 3D acoustic information with visual data. It proposes a 4D audio-visual framework that projects acoustic heatmaps from an acoustic camera onto dynamic RGB-D point clouds and uses a transformer-based Acoustic Event Detection module to identify relevant tool–tissue interactions. Event localization is performed with a DBSCAN-based clustering approach to produce 3D bounding boxes that align with the visual scene, enabling robust multimodal fusion. Evaluations in a realistic operating-room setup show accurate 3D sound localization and effective fusion, marking the first demonstration of spatial sound localization in dynamic surgical scenes and paving the way for multimodal, data-driven surgical systems.

Abstract

Purpose: Surgical scene understanding is key to advancing computer-aided and intelligent surgical systems. Current approaches predominantly rely on visual data or end-to-end learning, which limits fine-grained contextual modeling. This work aims to enhance surgical scene representations by integrating 3D acoustic information, enabling temporally and spatially aware multimodal understanding of surgical environments. Methods: We propose a novel framework for generating 4D audio-visual representations of surgical scenes by projecting acoustic localization information from a phased microphone array onto dynamic point clouds from an RGB-D camera. A transformer-based acoustic event detection module identifies relevant temporal segments containing tool-tissue interactions which are spatially localized in the audio-visual scene representation. The system was experimentally evaluated in a realistic operating room setup during simulated surgical procedures performed by experts. Results: The proposed method successfully localizes surgical acoustic events in 3D space and associates them with visual scene elements. Experimental evaluation demonstrates accurate spatial sound localization and robust fusion of multimodal data, providing a comprehensive, dynamic representation of surgical activity. Conclusion: This work introduces the first approach for spatial sound localization in dynamic surgical scenes, marking a significant advancement toward multimodal surgical scene representations. By integrating acoustic and visual data, the proposed framework enables richer contextual understanding and provides a foundation for future intelligent and autonomous surgical systems.

Figures (4)

• Figure 1: Conceptual overview of the proposed system with an example obtained from surgical chiseling. In this example, hammer events are detected by the Acoustic Event Detection stage, as described in \ref{['sec:eventdetection']}. Detected events trigger the Acoustic Event Localization stage, which projects the acoustic heatmap onto the dynamic 3D scene representation and localizes the event. We utilize point clouds from a co-calibrated RGB-D camera to represent the scene in our experiments, as described in \ref{['sec:localization']}. An optical tracking system is used for the evaluation of the system to compare the result with the ground truth pose of the surgical instrument.
• Figure 2: Overview of the experimental setup. We use our camera setup including the Ring48 microphone array, the ZED 2i RGB-D camera, and the FusionTrack 500 tracking system (shown in the left frame) to capture sequences of chiseling, drilling, and sawing (shown in the right frame).
• Figure 3: We illustrate qualitative results over one full example recording of length $L_{sequence} = 20s$, obtained from the experiment described in \ref{['sec:evaluation']}, for each of the surgical actions analyzed within this work. The detections have been obtained using the relaxed condition. The red arrows on top are the ground truth events, the green arrows on the bottom are the predicted events.
• Figure 4: Distribution of 3D bounding box IoU scores over a total of 20 recordings, grouped by surgical action.