Modular Autonomy with Conversational Interaction: An LLM-driven Framework for Decision Making in Autonomous Driving

TL;DR

The paper addresses the challenge of translating natural-language passenger intents into safe, actionable commands within a modular autonomous driving stack. It introduces an LLM-based interaction layer layered on Autoware, anchored by a five-category interaction taxonomy, an application-centric DSL, and a dedicated Validation/Interface Node, complemented by a two-stage feedback loop using a cloud LLM and TTS for transparent user communication. Empirical results show a translation accuracy of $97.0\%$ and a Task Success Rate of $96.7\%$ in simulated runs, with ablation studies highlighting the importance of examples and the safety-focused design. The work advances extensible, safety-conscious natural-language interfaces for modular ADS, laying groundwork for domain-adaptive reasoning and richer human–vehicle collaboration while highlighting cloud-dependency and broader-scenario evaluation needs.

Abstract

Recent advancements in Large Language Models (LLMs) offer new opportunities to create natural language interfaces for Autonomous Driving Systems (ADSs), moving beyond rigid inputs. This paper addresses the challenge of mapping the complexity of human language to the structured action space of modular ADS software. We propose a framework that integrates an LLM-based interaction layer with Autoware, a widely used open-source software. This system enables passengers to issue high-level commands, from querying status information to modifying driving behavior. Our methodology is grounded in three key components: a taxonomization of interaction categories, an application-centric Domain Specific Language (DSL) for command translation, and a safety-preserving validation layer. A two-stage LLM architecture ensures high transparency by providing feedback based on the definitive execution status. Evaluation confirms the system's timing efficiency and translation robustness. Simulation successfully validated command execution across all five interaction categories. This work provides a foundation for extensible, DSL-assisted interaction in modular and safety-conscious autonomy stacks.

Figures (6)

• Figure 1: Conceptual diagram of the -based co-pilot interfacing natural language instructions and requests with the modular software stack.
• Figure 2: Illustration of the proposed -based architecture for processing natural language spoken instructions into executable commands for an . The integrates status information, a contextual knowledge base, which defines the 's role and contains examples, and the Spoken Instruction. It then translates this input into a structured Extracted Command in a format. The Extracted Command is routed to a Interface Node for validation and subsequent Command Execution by the 's control systems. Successful execution triggers a confirmation, enabling appropriate Passenger Feedback.
• Figure 3: Integration architecture featuring the Validation and Interface Node, which connects the extracted command from the to the modular software stack, especially the Planning and Decision Making module, via .
• Figure 4: Boxplot comparison of system response times $t_r$ for the translation task. The results from the $N=200$ dataset for our proposed system () are compared against the same system with ablations and against the results from Song et al. Song2025 (), which used a different dataset of same size and a similar, but non-identical experimental setup with a different model and . Note on data extraction: The characteristic values for the boxplot representing Song2025 were derived using a digital plot-digitizer tool from the published figure, as the original numerical data were unavailable.
• Figure 5: Time-series plot illustrating the vehicle's velocity profile $v$ over time $t$ in the Autoware planning simulation environment, showcasing the execution of multiple passenger instructions. At $t_0$ - $t_5$ spoken instructions were issued by the system and executed by the software.