ADAS-TO: A Large-Scale Multimodal Naturalistic Dataset and Empirical Characterization of Human Takeovers during ADAS Engagement

TL;DR

ADAS-TO is presented, the first large-scale naturalistic dataset dedicated to ADAS-to-manual transitions, containing 15,659 takeover-centered 20s clips from 327 drivers across 22 vehicle brands, and finds that in 59.3% of critical cases, actionable visual cues emerge at least 3s before takeover, supporting the potential for semantics-aware early warning beyond late-stage kinematic triggers.

Abstract

Takeovers remain a key safety vulnerability in production ADAS, yet existing public resources rarely provide takeover-centered, real-world data. We present ADAS-TO, the first large-scale naturalistic dataset dedicated to ADAS-to-manual transitions, containing 15,659 takeover-centered 20s clips from 327 drivers across 22 vehicle brands. Each clip synchronizes front-view video with CAN logs. Takeovers are defined as ADAS ON $\rightarrow$ OFF transitions, with the primary trigger labeled as brake, steer, gas, mixed, or system disengagement. We further separate planned driver-initiated terminations (Ego) from forced takeovers (Non-ego) using a rule-based partition. While most events occur within conservative kinematic margins, we identify a long tail of 285 safety-critical cases. For these events, we combine kinematic screening with vision--language (VLM) annotation to attribute hazards and relate them to intervention dynamics. The resulting cross-modal analysis shows distinct kinematic signatures across traffic dynamics, infrastructure degradation, and adverse environments, and finds that in 59.3% of critical cases, actionable visual cues emerge at least 3s before takeover, supporting the potential for semantics-aware early warning beyond late-stage kinematic triggers. The dataset is publicly released at huggingface.co/datasets/HenryYHW/ADAS-TO-Sample.

Figures (16)

• Figure 1: Role of OP in data collection. (a) Device setup: Comma set up and OP performance (b) Passive logging under OEM ADAS: OP acts as an observer and captures vehicle dynamics and driver inputs around takeovers via CAN. (c) OP engaged (control): the system records its states and outputs from OP's driving model.
• Figure 2: Clip structure. Each 20 s clip is centered on the takeover event ($t = 0$).
• Figure 3: Dataset overview. (a) Top-10 vehicle brands by clip count (out of 22 total brands). (b) Distribution of vehicle speed at the takeover moment (dashed: mean; dash-dotted: median). (c) Distribution of following distance to lead vehicle (when detected, 49.8% of clips). (d) Primary takeover action distribution; the first action applied to take over the vehicle.
• Figure 4: Geographic distribution of the 2,244 routes with GPS coordinates across 323 unique drivers with 4 don't want to share the location. North America accounts for 84.2% of routes, followed by Europe (4.5%) and Asia (3.2%).
• Figure 5: Clips per driver. The distribution follows a long tail: the top-10 drivers (orange) contribute 40.4% of all clips.