Survey and synthesis of state of the art in driver monitoring

TL;DR

This paper addresses the problem of characterizing the driver state as the first step in driver monitoring (DM) and presents a structured, polychotomous framework linking five driver substates—drowsiness, mental workload, distraction, emotions, and under the influence—to indicators and sensors. It (i) surveys the DM/DMS literature up to early 2021, (ii) proposes a triad-based representation (states–indicators–sensors) and interlocked tables to synthesize state characterization, and (iii) details state-specific indicators and sensing modalities across driver, vehicle, and environment contexts. The work highlights that multimodal indicators are essential, that internal cameras and wearables are increasingly employed, and that many studies focus on present-state estimation rather than long-horizon prediction. It also emphasizes the need for explainable methods and a future research focus on forecasting driver state tens of seconds ahead to enable safer takeovers in SAE Levels $0$ to $5$, ultimately guiding researchers and industry in designing synergistic DM and driving-automation systems. The proposed tables serve as an at-a-glance reference to identify research gaps and opportunities for practical DM system development and standardization across automotive platforms. $6$ SAE levels of driving automation are considered, with DM remaining critical for safe operation at Levels $0$–$4$, and becoming unnecessary only at Level $5$ when the driver is never in control.

Abstract

Road-vehicle accidents are mostly due to human errors, and many such accidents could be avoided by continuously monitoring the driver. Driver monitoring (DM) is a topic of growing interest in the automotive industry, and it will remain relevant for all vehicles that are not fully autonomous, and thus for decades for the average vehicle owner. The present paper focuses on the first step of DM, which consists in characterizing the state of the driver. Since DM will be increasingly linked to driving automation (DA), this paper presents a clear view of the role of DM at each of the six SAE levels of DA. This paper surveys the state of the art of DM, and then synthesizes it, providing a unique, structured, polychotomous view of the many characterization techniques of DM. Informed by the survey, the paper characterizes the driver state along the five main dimensions--called here "(sub)states"--of drowsiness, mental workload, distraction, emotions, and under the influence. The polychotomous view of DM is presented through a pair of interlocked tables that relate these states to their indicators (e.g., the eye-blink rate) and the sensors that can access each of these indicators (e.g., a camera). The tables factor in not only the effects linked directly to the driver, but also those linked to the (driven) vehicle and the (driving) environment. They show, at a glance, to concerned researchers, equipment providers, and vehicle manufacturers (1) most of the options they have to implement various forms of advanced DM systems, and (2) fruitful areas for further research and innovation.

Figures (7)

• Figure 1: The flow diagram (1) illustrates the strategy used for our survey of the literature on driver monitoring (DM) and driver-monitoring systems (DMSs), and (2) shows the number of publications at each stage of the process.
• Figure 2: The figure shows, for the context of driver monitoring (DM), the system block diagram applicable to the characterization of a generic (sub)state. The input is the situation of interest and the output is the level of the state. The operation of each of the three subsystems is described in the text.
• Figure 3: The figure shows simplified representations of key Table \ref{['tab-indicators']} (states vs indicators) and Table \ref{['tab-sensors']} (sensors vs indicators). It also suggests that these tables can naturally be interpreted as being two views of an underlying 3D array. $\text{S}_{\text{i}}$, D, V, and E stand for “State i”, “Driver”, “Vehicle”, and “Environment", respectively.
• Figure 4: Graph (a) is the histogram of the number of references vs year for the $56$ initial references on driver monitoring (DM), and graph (b) is the corresponding histogram for the $254$ examined references. These histograms suggest that the field of DM has been the object of growing interest over the years and, in particular, over the last $10$ years.
• Figure :