MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments

TL;DR

MM-OR introduces the first large-scale, realistic multimodal operating room dataset and the first multimodal scene graph generation approach for ORs. The dataset aggregates RGB-D, high-resolution views, audio, transcripts, robotic logs, tracking data, and panoptic/scene-graph annotations across 92,983 timepoints from knee replacement procedures, totaling ~500 GB and synchronized at 1 FPS. The MM2SG model fuses modality-specific encoders with an LLM to generate structured scene graphs, with temporal memory and augmentation techniques that improve robustness; experiments show superior performance over baselines and clear value from each modality, especially for rare relations. Together, MM-OR and MM2SG establish a new benchmark for holistic, multimodal surgical scene understanding and enable downstream tasks such as robot phase prediction, action anticipation, and sterility breach detection in high-stakes environments.

Abstract

Operating rooms (ORs) are complex, high-stakes environments requiring precise understanding of interactions among medical staff, tools, and equipment for enhancing surgical assistance, situational awareness, and patient safety. Current datasets fall short in scale, realism and do not capture the multimodal nature of OR scenes, limiting progress in OR modeling. To this end, we introduce MM-OR, a realistic and large-scale multimodal spatiotemporal OR dataset, and the first dataset to enable multimodal scene graph generation. MM-OR captures comprehensive OR scenes containing RGB-D data, detail views, audio, speech transcripts, robotic logs, and tracking data and is annotated with panoptic segmentations, semantic scene graphs, and downstream task labels. Further, we propose MM2SG, the first multimodal large vision-language model for scene graph generation, and through extensive experiments, demonstrate its ability to effectively leverage multimodal inputs. Together, MM-OR and MM2SG establish a new benchmark for holistic OR understanding, and open the path towards multimodal scene analysis in complex, high-stakes environments. Our code, and data is available at https://github.com/egeozsoy/MM-OR.

Figures (6)

• Figure 1: Overview of a single timepoint in MM-OR, illustrating the multimodal data provided for each sample: RGB-D video from multiple angles, detailed RGB views, low-exposure video, point cloud data, robot screen and tracker logs, audio and speech transcripts, panoptic segmentations, semantic scene graphs, and downstream task annotations such as robot phase, next action, and sterility breach status.
• Figure 2: Recording setup and sensors overview. A grey circle by each sensor shows quantity; if absent, the sensor count is one.
• Figure 3: Overview of the proposed MM2SG architecture for multimodal scene graph generation. MM2SG processes a variety of data sources through specialized encoders, projecting them into a shared space. The language model generates scene graph triplets describing SGs with entities $E_i$ and predicates $p_i$. Downstream tasks leverage entire sequences of scene graphs rather than individual ones.
• Figure 4: Qualitative examples from a test take in MM-OR, illustrating scene graph generation performance of MM2SG. Unlabeled edges indicate the "close to" predicate.
• Figure 5: Qualitative segmentation results examples from a test take in MM-OR. The top row shows the ground truth segmentations and the bottom row shows the corresponding predictions.