Vision-language models for decoding provider attention during neonatal resuscitation

TL;DR

This work tackles the problem of quantifying provider gaze during neonatal resuscitation by introducing an automated, real-time semantic gaze classification pipeline that fuses real-time instance segmentation with vision-language models. Using a first-person NICU dataset (EDIR), the system demonstrates strong zero-shot performance when combining cropped gaze regions with segmentation, and substantial gains with few-shot and fine-tuned training, achieving Top-1 accuracy up to 93% and Top-3 accuracy up to 98% in single-label tasks, with high multi-label performance as well. Real-time inference across multiple GPUs is feasible, enabling objective quantification of gaze dynamics in live resuscitations and offering practical potential for training, decision support, and NICU design. The approach provides a scalable, data-efficient framework for semantic gaze analysis in high-stakes clinical environments and sets the stage for broader application across medical domains.

Abstract

Neonatal resuscitations demand an exceptional level of attentiveness from providers, who must process multiple streams of information simultaneously. Gaze strongly influences decision making; thus, understanding where a provider is looking during neonatal resuscitations could inform provider training, enhance real-time decision support, and improve the design of delivery rooms and neonatal intensive care units (NICUs). Current approaches to quantifying neonatal providers' gaze rely on manual coding or simulations, which limit scalability and utility. Here, we introduce an automated, real-time, deep learning approach capable of decoding provider gaze into semantic classes directly from first-person point-of-view videos recorded during live resuscitations. Combining state-of-the-art, real-time segmentation with vision-language models (CLIP), our low-shot pipeline attains 91\% classification accuracy in identifying gaze targets without training. Upon fine-tuning, the performance of our gaze-guided vision transformer exceeds 98\% accuracy in gaze classification, approaching human-level precision. This system, capable of real-time inference, enables objective quantification of provider attention dynamics during live neonatal resuscitation. Our approach offers a scalable solution that seamlessly integrates with existing infrastructure for data-scarce gaze analysis, thereby offering new opportunities for understanding and refining clinical decision making.

Figures (4)

• Figure 1: Approach. (Left) During resuscitation, physicians must attend to multiple stimuli at once. (Middle) The output of Tobii eye-tracking glasses can be used to isolate the subject with segmentation (top) and cropping (bottom). (Right) Cropped images and object masks are then fed to the model for semantic gaze classification, and prediction scores are aggregated for each target for attention analysis. Note: depicted infant is synthetic.
• Figure 2: Sample gaze classification predictions on cropped (top) and segmented (bottom) testing images. Note: “CMAC-Screen” refers to "Video Laryngoscope Screen."
• Figure 3: Model class activation maps with GradCAM on the Laryngoscope Screen. Each heat map conveys where the model is “looking” in this example image, where each model correctly predicted the class label. Less accurate models, like the ResNet-50, have more diffuse heat maps whereas the higher-performing fine-tuned CLIP (middle) and the MobileViT (far-right) models have heat maps concentrated on the object of attention, the Laryngoscope screen.
• Figure 4: Automated pipeline captures neonatologist gaze dynamics.(A) Using the MobileViT model for semantic gaze classification, predicted visual attention approaches the expert-level annotations (ns: not significant --- i.e., our model makes predictions as accurately as our human annotators). (B). Visualizing gaze transitions between areas of interest. Transition matrix probabilities should be read as directed arrows from labels on the left to labels on the bottom, and are normalized per-row. Non-transitions (self-loops along the diagonal) are reported but omitted from the colormap. Notable findings include the strong Vitals $\rightarrow$ Laryngoscope Screen transition, and the Infant $\leftrightarrow$ Airway-Provider cycle, which may be due to their spatial proximity. (C). Visualizing gaze transitions over the course of a session. Notable findings include the blocks of Airway Equipment early on (i.e., during placement), the Laryngoscope Screen block (i.e., during intubation), and the Vitals Monitor block (i.e., during patient stabilization).