Enhancing Fatigue Detection through Heterogeneous Multi-Source Data Integration and Cross-Domain Modality Imputation
Luobin Cui, Yanlai Wu, Tang Ying, Weikai Li
TL;DR
This work addresses fatigue detection in safety-critical settings where high-fidelity sensors are impractical by proposing a heterogeneous multi-source framework that transfers knowledge from deployment-constrained sources to a real-world target domain. It formalizes a cross-domain problem with modality heterogeneity, provides a theoretical generalization bound showing benefits from imputing missing sensors, and defines a learning objective that jointly optimizes sensor reconstruction, domain alignment, and task accuracy. The framework is validated on VPFD, MEFAR, and FatigueSet, with regression-based imputers (MLP) outperforming alternatives and BN plus Jacobian regularization enhancing performance when multiple imputed modalities are available. This approach demonstrates practical gains in fatigue detection under sensor-constrained conditions and offers a path toward robust, real-time deployments in diverse field environments.
Abstract
Fatigue detection for human operators plays a key role in safety critical applications such as aviation, mining, and long haul transport. While numerous studies have demonstrated the effectiveness of high fidelity sensors in controlled laboratory environments, their performance often degrades when ported to real world settings due to noise, lighting conditions, and field of view constraints, thereby limiting their practicality. This paper formalizes a deployment oriented setting for real world fatigue detection, where high quality sensors are often unavailable in practical applications. To address this challenge, we propose leveraging knowledge from heterogeneous source domains, including high fidelity sensors that are difficult to deploy in the field but commonly used in controlled environments, to assist fatigue detection in the real world target domain. Building on this idea, we design a heterogeneous and multiple source fatigue detection framework that adaptively utilizes the available modalities in the target domain while exploiting diverse configurations in the source domains through alignment across domains and modality imputation. Our experiments, conducted using a field deployed sensor setup and two publicly available human fatigue datasets, demonstrate the practicality, robustness, and improved generalization of our approach across subjects and domains. The proposed method achieves consistent gains over strong baselines in sensor constrained scenarios. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.