Autoware.Flex: Human-Instructed Dynamically Reconfigurable Autonomous Driving Systems

TL;DR

Autoware.Flex tackles the dual challenges of misinterpretation in complex driving scenarios and accommodating user-driving preferences by introducing a human-in-the-loop ADS. It translates natural language instructions into a domain-specific AutoIR representation via an LLM augmented with an ADS knowledge base using Retrieval-Augmented Generation, followed by a safety-driven rule-based validation that safely injects changes into Autoware.Universe. The approach is validated through simulations and real-vehicle experiments, demonstrating accurate NL-to-AutoIR translation, fast rule matching, and safe execution under varied scenarios. The work contributes AutoIR semantics, a specialized ADS knowledge base, a rule-base design with runtime validation, and practical demonstrations of user-guided but safety-conscious autonomous driving. This framework enables more context-aware and customizable ADS while maintaining reliable safety guarantees in real-world operation.

Abstract

Existing Autonomous Driving Systems (ADS) independently make driving decisions, but they face two significant limitations. First, in complex scenarios, ADS may misinterpret the environment and make inappropriate driving decisions. Second, these systems are unable to incorporate human driving preferences in their decision-making processes. This paper proposes Autoware$.$Flex, a novel ADS system that incorporates human input into the driving process, allowing users to guide the ADS in making more appropriate decisions and ensuring their preferences are satisfied. Achieving this needs to address two key challenges: (1) translating human instructions, expressed in natural language, into a format the ADS can understand, and (2) ensuring these instructions are executed safely and consistently within the ADS' s decision-making framework. For the first challenge, we employ a Large Language Model (LLM) assisted by an ADS-specialized knowledge base to enhance domain-specific translation. For the second challenge, we design a validation mechanism to ensure that human instructions result in safe and consistent driving behavior. Experiments conducted on both simulators and a real-world autonomous vehicle demonstrate that Autoware$.$Flex effectively interprets human instructions and executes them safely.

Figures (13)

• Figure 1: A complex scenario: traffic lights malfunction, and a traffic officer directs vehicles at the intersection.
• Figure 2: An overview of Autoware.Flex
• Figure 3: The workflow of user instruction translation
• Figure 4: The architectures of Autoware and ROS 2
• Figure 5: An example of an AutoIR program