Graph-based Online Monitoring of Train Driver States via Facial and Skeletal Features

TL;DR

The study addresses railway driver safety by moving beyond basic dead-man switches to a vision-based online monitoring system. It introduces a customised Directed Graph Neural Network (DGNN) that processes a directed whole-body graph constructed from fused facial and skeletal keypoints, enabling three-class classification into alert, not alert, and pathological states. An ablation study demonstrates that combining facial and skeletal cues yields the best performance, achieving 80.88% accuracy for three-class classification and over 99% for binary alertness, while a novel dataset with simulated pathological conditions under varying lighting conditions broadens evaluation. This approach enhances robustness to lighting changes and provides a pathway toward real-time, safety-critical monitoring in railway operations, with future work exploring attention-based architectures and thermal imaging.

Abstract

Driver fatigue poses a significant challenge to railway safety, with traditional systems like the dead-man switch offering limited and basic alertness checks. This study presents an online behavior-based monitoring system utilizing a customised Directed-Graph Neural Network (DGNN) to classify train driver's states into three categories: alert, not alert, and pathological. To optimize input representations for the model, an ablation study was performed, comparing three feature configurations: skeletal-only, facial-only, and a combination of both. Experimental results show that combining facial and skeletal features yields the highest accuracy (80.88%) in the three-class model, outperforming models using only facial or skeletal features. Furthermore, this combination achieves over 99% accuracy in the binary alertness classification. Additionally, we introduced a novel dataset that, for the first time, incorporates simulated pathological conditions into train driver monitoring, broadening the scope for assessing risks related to fatigue and health. This work represents a step forward in enhancing railway safety through advanced online monitoring using vision-based technologies.

Figures (6)

• Figure 1: Flowchart of our monitoring system framework. The inputs are the images acquired by an RGB camera, from which facial and skeletal features are extracted. These keypoints are used to construct a whole-body graph, which is then processed by a custom DGNN to classify the user's state into three categories: alert, not alert, or potentially experiencing a pathological condition.
• Figure 2: Samples of the defined directed graphs, which correspond to the input of our network, in the three tested user's states: alert, not alert, and pathological. The joints are shown in blue, and the bones connecting the joints are represented in red.
• Figure 3: The experimental setup consisted of users seated at a desk, following on-screen instructions to manipulate control levers, press dashboard buttons, and check side panels. An RGB camera continuously monitored the subject from the front. We tried to replicate, as closely as possible, the driving cabs configuration of rail vehicles and distances between the train driver and the dashboard specified in CabinReport.
• Figure 4: Comparison of test accuracies across four network variants, denoted by different colors: orange refers to the proposed DGNN for binary state classification using the whole-body graph (considering both skeletal and facial keypoints); red corresponds to the proposed three-class DGNN model using the whole-body graph; blue indicates the model using when only facial features, and green represents the model using only skeletal features.
• Figure 5: Confusion matrices for the proposed three-class model considering whole-body keypoints under the three different lighting conditions.