EgoEMS: A High-Fidelity Multimodal Egocentric Dataset for Cognitive Assistance in Emergency Medical Services

TL;DR

EgoEMS tackles the cognitive burden faced by EMS responders by providing the first end-to-end, high-fidelity, multimodal egocentric dataset of multiperson EMS workflows, aligned with national protocols. The dataset combines egocentric video, audio transcripts with speaker diarization, smartwatch IMU data, and ground-truth CPR metrics across 233 simulated trials and 62 participants, with detailed 67 keysteps across 9 interventions. It introduces an EMS-specific taxonomy and a semi-automatic annotation pipeline, plus three benchmarks for keystep classification, segmentation, and CPR quality estimation to drive real-time cognitive assistance research. The work emphasizes privacy-preserving data collection and release, open-source tools, and a path toward responsible real-world deployment, including governance and transfer-learning considerations. Overall, EgoEMS provides a rich, replicable resource to develop AI systems that support EMS decision-making and potentially improve patient outcomes in critical emergencies.

Abstract

Emergency Medical Services (EMS) are critical to patient survival in emergencies, but first responders often face intense cognitive demands in high-stakes situations. AI cognitive assistants, acting as virtual partners, have the potential to ease this burden by supporting real-time data collection and decision making. In pursuit of this vision, we introduce EgoEMS, the first end-to-end, high-fidelity, multimodal, multiperson dataset capturing over 20 hours of realistic, procedural EMS activities from an egocentric view in 233 simulated emergency scenarios performed by 62 participants, including 46 EMS professionals. Developed in collaboration with EMS experts and aligned with national standards, EgoEMS is captured using an open-source, low-cost, and replicable data collection system and is annotated with keysteps, timestamped audio transcripts with speaker diarization, action quality metrics, and bounding boxes with segmentation masks. Emphasizing realism, the dataset includes responder-patient interactions reflecting real-world emergency dynamics. We also present a suite of benchmarks for real-time multimodal keystep recognition and action quality estimation, essential for developing AI support tools for EMS. We hope EgoEMS inspires the research community to push the boundaries of intelligent EMS systems and ultimately contribute to improved patient outcomes.

Figures (39)

• Figure 1: EgoEMS dataset provides synchronized egocentric multiperson views, along with rich high-fidelity multimodal data capturing EMS professionals in highly procedural tasks resulting in a total of 233 trials (20 hours) including 2694 keystep instances from commonly executed EMS interventions. The dataset includes annotations of keysteps, timestamped audio transcripts with speaker diarization, and semi-automatically generated bounding boxes and segmentation masks for key EMS objects, offering a comprehensive resource for understanding EMS workflows and developing AI solutions.
• Figure 2: Methodology for creating the EMS taxonomy and EgoEMS simulation scenarios.
• Figure 3: Top-10 keystep distribution with average duration.
• Figure 4: Ground-truth average CPR compression rate per participant across three skill levels (Public, Member, EMT).
• Figure 5: Overview of benchmark tasks for keystep classification, segmentation, and CPR quality estimation using EgoEMS data. Example baseline models and representative results highlight the diverse evaluation scenarios enabled by the dataset.