TeamVision: An AI-powered Learning Analytics System for Supporting Reflection in Team-based Healthcare Simulation

TL;DR

TeamVision presents an AI-powered multimodal analytics system that captures voice, transcripts, body rotation, and positioning to guide debriefs after healthcare simulations. Built through five educators and a human-centered design process, it was evaluated in-the-wild with 56 nursing teams and 221 students, providing an educator-facing dashboard and a shared debrief view. Findings indicate that TeamVision can structure and tailor reflective discussions, particularly around communication, teamwork, and task management, but caution is warranted due to data accuracy and trust concerns, especially for group-level visuals. The work advances HC-AI design by demonstrating real-world deployment of multimodal analytics in education and offers actionable insights for integrating AI-assisted reflection tools into team-based healthcare training.

Abstract

Healthcare simulations help learners develop teamwork and clinical skills in a risk-free setting, promoting reflection on real-world practices through structured debriefs. However, despite video's potential, it is hard to use, leaving a gap in providing concise, data-driven summaries for supporting effective debriefing. Addressing this, we present TeamVision, an AI-powered multimodal learning analytics (MMLA) system that captures voice presence, automated transcriptions, body rotation, and positioning data, offering educators a dashboard to guide debriefs immediately after simulations. We conducted an in-the-wild study with 56 teams (221 students) and recorded debriefs led by six teachers using TeamVision. Follow-up interviews with 15 students and five teachers explored perceptions of its usefulness, accuracy, and trustworthiness. This paper examines: i) how TeamVision was used in debriefing, ii) what educators found valuable and challenging, and iii) perceptions of its effectiveness. Results suggest TeamVision enables flexible debriefing and highlights the challenges and implications of using AI-powered systems in healthcare simulation.

Figures (6)

• Figure 1: TeamVision: An AI-powered multimodal system that captures (A) human observations, real-time positioning, and audio data from nursing students during high-fidelity clinical simulations. It provides an analytics dashboard. (B) Educators can filter and customise data visualisations to fit their approach, while (C) students view performance analytics on a shared screen during the debrief.
• Figure 2: Initial designs considered during the workshop design sessions. A) a priority chart depicting the team's prioritisation behaviours (orange bar) and the values from previous high-performing teams. B) Speech and location ward map illustrating the amount of speaking time in a particular location. C) Speech sociogram showing the amount of speaking time per student and their interactions with other students/roles.
• Figure 3: Catalogue of visualisations and sources in the debrief control view: A) Priority chart, B) Speech and location ward map, C) Speech sociogram, D) Communication network, and E) Video snippet.
• Figure 4: (A) Tagging and annotation view at the personal device during observation of clinical scenario. Educators can choose from (1) main phases or (2) actions. B) Debrief control view on the personal device. The main features are (3) the timeline to control the selection and navigation of data, (4) a quick selection of main phases, (5) a catalogue of visualisations, and (6) the option to share a visualisation to the shared screen.
• Figure 5: Top: Strategies followed by educators while using the filtering option during their debrief sessions. Note how educators mostly used only the information of one particular phase to generate their corresponding visualisations and support their discussions. Bottom: Use frequency of visualisations per phase. The communication network and sociogram were frequently used together when visiting all phases.