Visual and Cognitive Demands of a Large Language Model-Powered In-vehicle Conversational Agent

TL;DR

This study examines the visual and cognitive demands of an LLM-powered in-vehicle conversational agent (Gemini Live) during on-road driving, using a five-task battery (Visual Turn-by-turn, Hands-Free Phone Call, Gemini Live Single-turn, Gemini Live Multi-turn, and OSPAN) and objective measures (DRT, eye-tracking) plus subjective workload ratings. Across 32 participants, Gemini Live induced cognitive load between visual turn-by-turn guidance and the high-load OSPAN, while visual demand remained low with mean glances well under the 2-second threshold; single-turn interactions often yielded the lowest overall demand in TEORT. Multi-turn Gemini Live conversations showed stable cognitive load over time, with no cumulative increases in DRT metrics, suggesting resilient performance during extended dialogue. Subjective responses aligned with objective data, with Gemini Live rated highly for usability and satisfaction, supporting the safe deployment of voice-based, LLM-powered agents in vehicles under the tested conditions, though real-world variability and connectivity issues warrant cautious generalization and further study.

Abstract

Driver distraction remains a leading contributor to motor vehicle crashes, necessitating rigorous evaluation of new in-vehicle technologies. This study assessed the visual and cognitive demands associated with an advanced Large Language Model (LLM) conversational agent (Gemini Live) during on-road driving, comparing it against handsfree phone calls, visual turn-by-turn guidance (low load baseline), and the Operation Span (OSPAN) task (high load anchor). Thirty-two licensed drivers completed five secondary tasks while visual and cognitive demands were measured using the Detection Response Task (DRT) for cognitive load, eye-tracking for visual attention, and subjective workload ratings. Results indicated that Gemini Live interactions (both single-turn and multi-turn) and hands-free phone calls shared similar levels of cognitive load, between that of visual turn-by-turn guidance and OSPAN. Exploratory analysis showed that cognitive load remained stable across extended multi-turn conversations. All tasks maintained mean glance durations well below the well-established 2-second safety threshold, confirming low visual demand. Furthermore, drivers consistently dedicated longer glances to the roadway between brief off-road glances toward the device during task completion, particularly during voice-based interactions, rendering longer total-eyes-off-road time findings less consequential. Subjective ratings mirrored objective data, with participants reporting low effort, demands, and perceived distraction for Gemini Live. These findings demonstrate that advanced LLM conversational agents, when implemented via voice interfaces, impose cognitive and visual demands comparable to established, low-risk hands-free benchmarks, supporting their safe deployment in the driving environment.

Figures (7)

• Figure 1: Route in Bowie, MD.
• Figure 2: View taken from the Tobii Pro 3's forward-facing camera.
• Figure 3: An Exponent employee with the DRT tactor and Tobii Pro 3 Glasses.
• Figure 4: DRT estimated marginal means and 95% Confidence Intervals.
• Figure 5: Gemini Live Multi-turn duration analysis results.