Robotic Classification of Divers' Swimming States using Visual Pose Keypoints as IMUs

TL;DR

This work addresses the challenge of monitoring underwater divers for life-threatening events without relying on diver-worn sensors or unreliable wireless links. It combines monocular vision-based 3D pose estimation (YOLOv8 + VideoPose3D) with a torso-centered body frame to generate pseudo-IMU features capturing translational and rotational accelerations, then applies multiple time-series classifiers to detect transitions from swimming to not-swimming. The study benchmarks six TSC models, demonstrates that translational features provide the strongest signal, and proves feasibility with an onboard AUV implementation and open-water tests, while noting limitations due to pose-estimation transfer across domains. The approach offers a practical path to real-time, robot-assisted diver safety, reducing dependency on on-body instrumentation and mitigating communication bottlenecks underwater, with ongoing work toward multimodal distress indicators and broader field trials.

Abstract

Traditional human activity recognition uses either direct image analysis or data from wearable inertial measurement units (IMUs), but can be ineffective in challenging underwater environments. We introduce a novel hybrid approach that bridges this gap to monitor scuba diver safety. Our method leverages computer vision to generate high-fidelity motion data, effectively creating a ``pseudo-IMU'' from a stream of 3D human joint keypoints. This technique circumvents the critical problem of wireless signal attenuation in water, which plagues conventional diver-worn sensors communicating with an Autonomous Underwater Vehicle (AUV). We apply this system to the vital task of identifying anomalous scuba diver behavior that signals the onset of a medical emergency such as cardiac arrest -- a leading cause of scuba diving fatalities. By integrating our classifier onboard an AUV and conducting experiments with simulated distress scenarios, we demonstrate the utility and effectiveness of our method for advancing robotic monitoring and diver safety.

Figures (10)

• Figure 1: Temporal classification of scuba diver swimming state conducted during a closed-water evaluation of the diver anomaly classification system. The system is deployed on an AUV. The AUV provides visual feedback of predicted state by illuminating a series of concentric LED lights controlled to reflect a green color which indicates the diver is swimming.
• Figure 2: Typical diver poses during scuba diving operations. Deep neural networks trained on terrestrial data often fail to localize human joints for these poses, given the uniqueness of the joint configurations and visibility.
• Figure 3: (Best if viewed at $3\times$ zoom level). Experimental setup for in-water data collection of non-standard body pose data and diver state transition data. The trackline delineates distance between the camera and participant. We collected image data at a depth of approximately $3.5$ m and between $3$ to $5$ m from the camera's image plane.
• Figure 4: (Best if viewed at $1.5\times$ zoom level). Pose keypoints labeled using an augmented COCO convention. We utilize the keypoints highlighted in orange in our analysis.
• Figure 5: Example moving and not moving images. In this case, the diver is hovering prone down, and when they stop moving they sink.