Real-Time In-Cabin Driver Behavior Recognition on Low-Cost Edge Hardware

TL;DR

This work presents a deployable, single-camera driver behavior recognition system for real-time edge deployment on two low-cost platforms (Raspberry Pi 5 and Coral Edge-TPU). It combines a compact per-frame classifier with a confounder-aware 17-class taxonomy and a temporal decision head that yields stable event-level alerts by enforcing confidence and persistence through a windowed gating scheme with parameters $K$ and $\tau$. The system is trained on a large, diverse in-cabin dataset (~800k images) with a driver- and vehicle-disjoint test protocol and validated in real-vehicle trials, achieving about $16$ FPS on Raspberry Pi 5 and $25$ FPS on Coral while maintaining low-latency alerts. These results demonstrate a practical sensing foundation for real-time, privacy-conscious driver monitoring and can support higher-level, human-centered vehicle intelligence models, including emerging agentic vehicle concepts.

Abstract

In-cabin Driver Monitoring Systems (DMS) must recognize distraction- and drowsiness-related behaviors with low latency under strict constraints on compute, power, and cost. We present a single-camera in-cabin driver behavior recognition system designed for deployment on two low-cost edge platforms: Raspberry Pi 5 (CPU-only) and Google Coral Edge TPU. The proposed pipeline combines (i) a compact per-frame vision model, (ii) a confounder-aware label design to reduce visually similar false positives, and (iii) a temporal decision head that triggers alerts only when predictions are both confident and sustained. The system covers 17 behavior classes, including multiple phone-use modes, eating/drinking, smoking, reaching behind, gaze/attention shifts, passenger interaction, grooming, control-panel interaction, yawning, and eyes-closed sleep. Training and evaluation use licensed datasets spanning diverse drivers, vehicles, and lighting conditions (details in Section 6), and we further validate runtime behavior in real in-vehicle tests. The optimized deployments achieve about 16 FPS on Raspberry Pi 5 with INT8 inference (per-frame latency under 60 ms) and about 25 FPS on Coral Edge TPU, enabling real-time monitoring and stable alert generation on inexpensive hardware. Finally, we discuss how reliable in-cabin human-state perception can serve as an upstream input for human-centered vehicle intelligence, including emerging agentic vehicle concepts.

Figures (2)

• Figure 1: End-to-end driver behavior alert pipeline (conceptual).
• Figure 2: Prototype edge-hardware setups used for live in-vehicle evaluation (camera + compute unit).